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The  Nominal  Group  Technique  (NGT)  .was  developed  as  an  alternative 
group  decision  technique  to  other  group  methods  such  as  the  Interactive, 
Brainstorming,  and  Delphi  techniques.    Although  several  past  studies 
have  shown  the  NGT  to  be  superior  to  other  group  methods,  it  was 
hypothesized  that  the  NGT  could  be  improved.    Specifically,  this  study 
examines  two  aspects  of  the  NGT.    The  first  is  a  modification  to 
the  NGT  process  (Mod  NGT)  in  which  the  verbalization  of  ideas  is  re- 
placed by  an  anonymous  written  system  utilizing  3x5  cards  in  an  at- 
tempt to  avoid  the  "linking"  of  ideas  with  individuals.    A  second 
factor  considered  was  whether  the  performance  of  the  NGT  is  affected 
by  whether  the  Nominal  Group  question  addresses  an  issue  that  is 
relatively  sensitive  (or  nonsensiti ve)  to  the  participants. 

The  NGT  question  was  a  problem  solution  task  involving  possible 
ways  of  improving  the  Army  ROTC  program.    The  research  design  was  a 
completely  randomized  field  experiment  using  experimental  and  control 


vi 


groups.    One  hundred  and  forty-four  Army  ROTC  cadets  were  randomly 
assigned  to  24,  6  member,  ad  hoc  groups.    Dependent  measures  were  the 
perceived  level  of  group  satisfaction,  the  number  of  ideas  withheld 
from  the  group,  the  quantity  of  ideas  produced,  and  a  consensual  judge- 
ment rating  of  the  "top-seven"  ideas  generated.    Interval  data  (number 
of  ideas  withheld  and  the  quantity  of  ideas)  were  analyzed  using 
Univariate  ANOVA  while  ordinal  data  (satisfaction  and  consensual 
judgement  rating)  were  analyzed  using  the  Kruskal-Wallis  Nonparametric 
ANOVA. 

The  overall  conclusions  were  that  the  Mod  NOT  groups  tended  to 

groups 

outperform  the  control  gropus  (using  the  Delbecq  NOT)  in  all  cases, 
whether  a  sensitive  or  nonsensitive  question  was  used.    It  was  also 
concluded  that  the  Delbecq  NGT  (control  groups)  tended  to  perforin 
best  using  a  nonsensitive,  rather  than  a  sensitive  question. 

Most  groups,  to  some  extent,  withheld  a  portion  of  their  ideas 
from  presentation.    The  Mod  NGT  groups  outperformed  the  Delbecq  NGT 
groups  by  a  three  to  one  margin,  with  the  latter  withholding  an 
average  of  more  than  nine  ideas  per  group. 


I.     INTRODUCTION  TO  THE  STUDY 


Group  decision  making  is  becoming  increasingly  common  in  modern 
organizations.    Tillman  (1960)  found  that  94  percent  of  firms  with 
more  than  10,000  employees  and  64  percent  with  less  than  250  employees 
utilized  formal  committees.    Van  de  Ven  (1974)  cites  numerous  studies 
which  have  concluded  that  executives  spend  an  average  time  of  between 
25  and  80  percent  of  their  time  in  committee  or  group  meetings.    He  also 
notes,  "A  characteristic  concern  of  contemporary  administrators  is  to 
find  viable  methods  for  making  decisions  where  a  number  of  people, 
differentiated  in  backgrounds  and  perspectives,  need  to  be  involved  in 
the  problem  solving  process"  (p.  1).    One  such  technique  for  group 
decision  making  is  the  Delbecq  et  al .  (1975)  Nominal  Group  Technique 
(NGT,  See  Appendix  A  for  a  detailed  description). 

The  NGT,  however,  is  not  the  only  technique  for  group  decision 
making.    In  fact  it  is  a  relatively  new  method  that  has  only  recently 
been  utilized  to  try  to  solve  some  of  the  "problems"  inherent  in  the 
most  widely  accepted  method  of  group  decision  making,  the  "Interactive 
Method."    Basically  the  interactive  method  is  defined  as  a  group  meeting 
in  which  all  communication  acts  take  place  between  members  with  minimal 
controls  or  formal  structuring  (Delbecq,  1968).    The  process  of  decision 
making  in  the  interacting  group  is  (1)  unstructured  group  discussion  for 
obtaining  and  pooling  ideas  of  the  participants  and  (2)  majority  voting 
on  priorities  by  hand  count  (Van  de  Ven,  1974). 
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Most  operating  managers  are  all  too  aware  of  some  of  the  dif- 
ficulties that  arise  from  the  use,  or  over  use  of  interactive  group 
meetings.    Many  meetings  fail  to  resolve  problems  and  participants 
often  become  frustrated  because  these  meetings  can  be  dominated  by 
power  groups,  they  can  get  bogged  down  with  argumentation  and  they 
seldom  leave  room  for  creative  thinking  and  suggestions.  Dunnette 
et  al.  (1963)  and  Campbell  (1968)  showed  that  they  can  be  restrictive 
and  dysfunctional  for  many  types  of  problem  solving.    Gustafson  et  al. 
(1973,  p.  282)  cite  many  reasons  why  interactive  groups  can  be  restrictive 
including: 

1 .  A  central  tendency  effect  where  groups  tend  to 
pursue  a  limited  train  of  thought  (Taylor  et  al . , 
1958;  Dunnette  et  al . ,  1963). 

2.  The  self  weighting  effect  where  the  individual  will 
participate  or  seek  to  exercise  influence  to  the 
extent  that  he  feels  equally  competent  with  others 
(Kelly  &  Thibaut,  1954;  Collaros  and  Anderson,  1969). 

3.  The  hidden  agenda  effect,  where  covert  judgements 

are  made  but  not  expressed  (Collaros  &  Anderson,  1969). 

4.  Group  Pressures  for  Conformity  and  the  implied  threats 
of  sanctions  from  the  more  knowledgeable  members 
(Dalkey  &  Hilmer,  1963;  Hoffman,  1965). 

5.  The  influence  of  dominant  personalities  (Dalkey  and 
Hilmer,  1963). 

6.  The  amount  of  time  and  effort  spent  by  the  group  to 
maintain  itself  (Dalkey  &  Hilmer,  1963;  Dunnette,  1964); 
and 

7.  A  tendency  to  reach  "speedy  decisions"  before  all  problem 
dimensions  have  been  considered  (Maier  &  Hoffman,  1960). 

A  complete  review  of  the  potential  problems  associated  with  inter- 
acting groups  is  outside  the  scope  of  this  dissertation.    Van  de  Ven  and 
Delbecq  (1971)  provide  an  excellent  review  of  these  problems. 

As  researchers  and  theorists  became  aware  of  the  widespread  use 
of  committees  and  the  problems  associated  with  interactive  groups, 
alternative  forms  were  proposed.    One  of  the  first  was  "Brainstorming" 
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(Osborne,  1963).    Brainstorming  is  a  relatively  unstructured  group 
process  which  is  designed  to  "free  individuals  from  inhibition,  self- 
criticism,  and  criticism  from  others  in  order  that,  in  response  to 
a  specified  problem,  they  may  produce  as  many  different  ideas  as  pos- 
sible" (Taylor  et  al.,  1958,  p.  24).    Although  Brainstorming  showed 
promise,  several  studies  (Bouchard,  1969;  Dunnette  et  al . ,  1963  and 
Taylor  et  al.,  1958)  have  demonstrated  that  group  participation  during 
brainstorming  inhibits  creative  thinking.    Another  difficulty  with  the 
process  is  there  is  no  evaluation,  voting,  or  ranking  phase  so  individuals 
can  leave  the  meeting  without  a  sense  of  closure. 

Another  alternative  to  the  interactive  group  is  the  Delphi 
technique,  which  was  developed  by  Norman  Dalkey  of  the  Rand  Corporation. 
The  Delphi  technique  is  designed  to  systematically  solicit  and  collate 
the  judgements  of  many  geographically  separated  individuals  on  a  specific 
topic.    It  utilizes  a  carefully  designed  set  of  questionnaires  inter- 
spersed with  summarized  information  and  feedback  of  opinions  derived 
from  earlier  responses.    The  Delphi  technique  does  solve  some  of  the 
problems  with  the  interactive  technique  but  since  one  topic  can  take 
as  much  as  five  months  to  complete,  its  usefulness  is  somewhat  limited. 
In  addition,  it  does  not  provide  the  productive  stimulation  of  positive 
face-to-face  interaction. 

The  most  recently  developed  alternative  method,  the  Nominal  Group 
Technique,  was  developed  by  Andre  Delbecq  and  Andrew  Van  de  Ven  in 
1968  as  one  way  to  avoid  the  many  problems  associated  with  interactive 
groups.    In  the  truest  sense,  the  use  of  the  term  "Nominal"  is  not 
totally  accurate  in  the  context  of  the  Delbecq  et  al .  (1975)  NGT 
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(hereafter  called  the  Delbecq  NGT,  the  Delbecq  Technique  or  the  NGT). 
Nominal  groups  are  usually  so  designated  because  they  are  groups  "in 
name  only"  as  verbal  exchange,  a  requirement  for  group  behavior,  is 
excluded  (Delbecq  et  al . ,  1975).    The  Delbecq  NGT -does  include  some 
provisions  for  interaction  among  group  members  but  arguments  and  lengthy 
discussions,  so  common  in  most  interactive  meetings,  are  specifically 
prohibited. 

The  NGT  is  a  special  purpose  technique  that  is  useful  in  situa- 
tions where  individual  judgements  must  be  tapped  and  combined  to  arrive 
at  decisions  which  cannot  be  calculated  by  one  person.    It  is  a  problem 
solving  or  an  idea  generation  technique  and  it  is  not  designed  for 
routine  meetings,  coordination,  bargaining  or  negotiations  (Delbecq 
et  al.,  1975).    Possible  suggested  uses,  other  than  traditional  group 
decision  making,  include:    as  a  training  tool  (Green,  1974);  as  an 
organizational  development  intervention  technique  (Mosley  &  Green, 
1974);  to  improve  upward  communications  (Green  &  Pietri,  1974);  as  a 
way  of  identifying  critical  health  problems  (Delbecq  et  al . ,  1972); 
as  a  pilot  research  technique  (Van  de  Ven  &  Delbecq,  1972);  and  as  part 
of  the  program  planning  process  (Delbecq  &  Van  de  Ven,  1971). 

The  NGT  is  designed  to  facilitate  creative  decision  making  where 
heterogeneous  group  members  must  pool  their  judgements  to  invent  or 
discover  a  satisfactory  course  of  action.    Thus  a  critical  requirement 
for  effective  leadership  is  to  ensure  that  judgemental  decisions  are 
facilitated  by  judgemental  techniques. 

The  NGT  is  a  structured  group  meeting  which  proceeds  according  to 
the  following  general  format: 
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1.  Silent  generation  of  ideas  by  the  participants  in 
writing  (a  form  of  silent  brainstorming). 

2.  Round  Robin  feedback  from  participants  of  tersely 
worded  ideas  on  a  flip  chart. 

3.  Serial  discussion  of  each  idea  for  clarification 
and  individual  evaluation  only. 

4.  Secret  voting  by  participants  to  rank  order  the  ideas. 

5.  (Optional)  discussion  of  vote  and  final  voting. 

The  NGT  is  designed  to  reduce  or  eliminate  some  of  the  inhibiting 
factors  (as  previously  cited)  that  are  inherent  in  interacting  groups. 
Cooley  (1977,  p.  30-32)  cites  a  number  of  process  characteristics  that 
enhance  the  effectiveness  of  the  NGT.    They  include: 

1.  Stimulate  "creative  tension"  by  means  of  the  presence 
of  others,  the  silence,  and  the  evidence  of  activity. 
This  tension  is  important  for  individual  commitment  to 
the  search  process.    Thus  the  social  facilitation  of  the 
group  setting  is  retained  and  amplified  (Van  de  Ven  and 
Delbecq,  1974). 

2.  Avoid  evaluation  or  elaborating  comments  while  problem 
dimensions  are  being  generated  (Maier  &  Hoffman,  1960). 

3.  Provide  each  individual  time  and  opportunity  to  engage 
in  reflection  (search)  and  force  participants  to  record 
their  thoughts  (Dunnette,  1964;  Horowitz  &  Newman,  1964; 
Maier  and  Solem,  1952). 

4.  Avoid  the  dominance  of  group  output  by  strong 
personality  types  (Maier  &  Maier,  1957). 

5.  Prevent  premature  closure  to  the  alternative  search 
process  and  decision  making  (Bennette,  1955;  Maier  &  Hoffman, 
1960). 

6.  Allow  all  participants  to  share  in  the  opportunity 

for  influencing  the  direction  of  group  decision  outcome 
(Goldman  et  al.,  1961;  Pelz,  1956). 

7.  Encourage  the  generation  of  minority  opinions  and  ideas 
which  consequently  are  more  likely  to  be  voiced  (Maier  & 
Solem,  1952;  Shukla,  1970). 

8.  Tolerate  conflicting,  incompatible  ideas  since  all  ideas 
are  revealed  in  writing  (Deutsch,  1949;  Guetzkow  &  Gyr, 
1954;  Vroom  et  al.,  1969). 

9.  Alleviate  "hidden  agendas"  or  covert  political  group 
dynamics  which  are  difficult  to  develop  when  writing 
(Foureizos  et  al.,  1950). 
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10.  Induce  a  sense  of  responsibility  in  the  members  for 
achieving  group  success  (Benne  &  Sheats,  1948). 

11.  Impose  a  burden  upon  all  participants  to  work  and 
produce  their  share  of  the  necessary  tasks  (Bales, 
1953;  Duetsch,  1949). 

12.  By  means  of  written  expression,  induce  a  greater 
feeling  of  commitment  and  a  greater  sense  of 
permanence  (Bouchard,  1969;  Horowitz  &  Newman,  1964). 

Mosely  and  Green  (1974)  succinctly  summarized  the  advantages  of  the 

Nominal  Group  Technique  by  noting  it  was  faster,  less  expensive,  and 

involved  more  people  in  a  shorter  time  frame  than  most  interactive 

meetings. 

Despite  these  process  advantages,  there  are  some  restrictions 
associated  with  the  NGT.    They  include: 

1.  Extended  preparation  for  the  NGT  meeting  is  usually 
necessary  to  identify  the  information  desired  from  a 
group  and  to  provide  the  necessary  supplies. 

2.  Inflexibility  of  the  structured  NGT  format  makes  it 
difficult  to  make  adjustments  or  to  change  topics 
in  the  middle  of  a  meeting;  therefore,  NGT  is 
generally  limited  to  a  single  purpose,  single  topic 
meeting.    There  are  some  situations  however,  (e.g.. 
Program  Planning,  and  Budget  and  Proposal  Review) 
where  a  series  of  related  topics  can  be  discussed 
using  the  NGT  format. 

3.  Conformity  behavior  to  a  structured  format  is 
required  on  the  part  of  all  participants,  a  condition 
which  is  not  immediately  comfortable  to  inexperienced 
participants . 

A  survey  of  recent  research  has  revealed  a  great  deal  of  evidence 
that  supports  the  conclusion  that  the  NGT  is  more  effective  than  the 
more  traditional  Interactive  method.    Several  authors  have,  however, 
cited  instances  where  the  NGT  has  proven  to  be  no  more  effective  than 
its  traditional  counterpart  and  still  others  have  suggested  some  modifi- 
cations to  the  technique  which  might  improve  its  effectiveness.  The 
most  relevant  empirical  studies  are  discussed  in  the  next  chapter. 


II.    REVIEW  OF  RELEVANT  EMPIRICAL  STUDIES 


A  review  of  the  literature  was  undertaken  to  examine  studies  that 
appeared  to  attack  issues  that  were  relevant  to  this  project.  The 
questions  addressed  and  sometimes  answered  in  these  works  provided 
additional  hypotheses  and  motivation  for  the  experimental  research 
undertaken  in  this  study. 

1.  Chung  and  Ferris  (1971)  compared  the  Nominal  Group  Technique 
with  the  Interactive  Group  method.    Twelve  graduate  students  completed 
a  written  case  analysis  and  the  authors  concluded  that  the  NGT  was 
superior  to  the  Interactive  method  and  that  the  NGT  can  reduce  inhibi- 
tions and  improve  the  quality  of  the  group's  decision. 

2.  Gustafson  et  al .  (1973)  evaluated  four  methods  of  eliciting 
Bayesian  subjective  likelihood  ratio  estimates.    The  authors  compared 
the  performances  of  individuals,  Interacting  groups,  Delphi  groups 

and  Nominal  groups  (NGT).    Subjects  were  288  freshmen  at  the  University 
of  Wisconsin,  and  were  evenly  divided  into  72  groups,  each  with  four 
members.    Eighteen  of  the  groups  were  randomly  assigned  to  each  of 
the  four  treatments.    An  "approximation"  of  the  NGT  was  utilized 
which  followed  an  estimation-talk-estimation  format  and  an  "open  dis- 
cussion" was  allowed  during  the  "talk"  phase.    Even  though  this  "open 
discussion"  is  a  violation  of  the  prescribed  NGT  format  of  Delbecq  et  al. 
(1975),  the  authors  still  concluded  the  nominal  process  (as  modified) 
was  superior  to  all  others  in  this  judgment  task. 

3.  Van  de  Ven  and  Delbecq  (1974)  compared  the  effectiveness  of 
Nominal  (NGT),  Delphi  and  Interacting  groups.    Student  dormitory 
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counselors  were  assigned  to  60  heterogeneous  groups  of  7  members  each 
using  a  "stratified  random  sampling"  procedure.    The  task  was  defining 
the  job  activities  for  student  dormitory  counselors.    The  authors 
concluded  that  (1)  both  the  NGT  and  the  Delphi  technique  were  clearly 
superior  to  Interacting  groups,  and  (2)  NGT  groups  generated  more  items 
and  expressed  greater  group  satisfaction  than  either  the  Delphi  or 
Interacting  groups.    The  Delbecq  NGT  format  was  followed  in  this  study. 

4.  Green  (1975)  examined  the  hypothesis  that  Nominal  groups  were 
superior  to  Interacting  groups  for  problem  identification  tasks  regard- 
less of  the  leadership  style  employed  in  the  Interacting  groups 
(Permissive,  Democratic  or  Authoritarian).    The  study  used  70  Electronic 
Data  Processing  (E.D.P.)  student  volunteers.    Subjects  did  not  know 
each  other  well  and  were  randomly  assigned  to  fourteen,  five  member 
groups.    The  task  was  "In  your  EDP  course,  what  problems  have  you 
noted?"    The  authors  utilized  a  three  step  (Listing,  Recording,  Voting) 
NGT  process.    The  Serial  Discussion  for  Clarification  phase  (Phase 
three)  of  the  Delbecq  et  al.  (1975)  NGT  procedure  was  omitted  from  this 
study.    The  authors  concluded  there  were  no  significant  differences 
between  the  Nominal  and  Interacting  groups  in  either  the  quantity  or 
quality  of  ideas  generated.    The  variation  in  the  NGT  format  by  Green 
(1975)  discounts  this  study  as  a  true  test  of  the  Delbecq  NGT  format. 

5.  Carr  (1975),  as  cited  by  Cooley  (1977)  compared  Nominal,  Inter- 
acting and  Brainstorming  groups  with  the  pooled  output  of  individuals 
working  alone.    Seventy- two  college  students  were  divided  into  twelve 
groups  and  were  asked  to  suggest  solutions  to  assigned  problems.  Carr 
concluded  that  pooled  individual  work  was  superior  to  each  of  the  three 
groups  methods.    He  further  noted  Nominal  groups  produced  more  ideas 


9 


than  Interacting  groups  and  identified  more  difficult  problems  than 
Brainstorming  groups.    Performance  was  compared  on  the  basis  of  quantity 
and  quality  of  ideas.    Cooley  presented  no  data  on  which  NGT  format 
was  utilized. 

6.  Frederick  (1976)  compared  Nominal  and  Interacting  groups  on 
a  problem  analysis  task.    Undergraduate  management  students  were 
randomly  assigned  to  twenty  groups  with  seven  members  each.    The  group 
task  was  to  indicate  any  barriers  that  might  prevent  the  group  from 
attaining  a  stated  goal.    Frederick  concluded  (1)  nominal  groups  did 
not  produce  a  greater  quantity  or  greater  number  of  high  quality 
ideas,  (2)  the  nominal  groups  did  produce  ideas  of  a  higher  average 
quality  than  Interacting  groups,  and  (3)  nominal  groups  produced  a 
greater  number  of  high  satisfaction  and  a  smaller  number  of  low  satis- 
faction individuals  than  did  the  interacting  groups.    No  information 
was  provided  on  which  NGT  format  was  utilized. 

7.  Nemiroff  et  al .  (1976)  compared  the  performance  of  Nominal, 
Consensual  and  Interacting  groups.    Consensual  groups  are  defined 

as  groups  employing  the  technique  whereby  each  member  must  agree  upon 
each  issue  or  idea.    Undergraduate  management  student  volunteers  were 
split  into  forty-eight,  four  member  groups.    The  task  was  a  "Lost  at 
Sea"  problem  with  an  objective  solution.    A  modified  Nominal  process 
was  utilized.    After  the  listing  phase,  the  round  robin  provided  that 
each  participant  must  present  all  of  his  rankings  at  one  time  and  then 
give  a  five  minute  rationale  for  his  selections.    After  the  five 
minute  rationale,  the  next  participant  listed  his  rankings  and  so  on 
until  each  member  had  spoken.    There  was  then  an  "open  discussion" 


phase  and  a  final  voting.    A  recorder  is  not  assigned  to  each  group. 
The  authors  concluded  that  (1)  consensual,  (modified)  Nominal  and 
Interacting  groups  ranked  one,  two,  three  on  decision  quality, (2) 
there  were  no  significant  differences  in  satisfaction  levels  between 
the  three  treatments  and  (3)  the  modified  NGT  consumed  significantly 
less  time  in  reaching  a  decision  than  the  other  two  methods. 

8.  Stead  (1976)  compared  the  performance  of  Nominal  and 
Sequential  Brainstorming  groups.    Sequential  Brainstorming  is  a 
structured  group  technique  that  utilizes  a  round  robin  format  for 
presenting  ideas.    Eighty-eight  state  employees  were  divided  into  six- 
teen groups  whose  sizes  ranged  from  four  to  seven  persons.    The  task 
was  "How  can  communications  be  improved  between  state  staff,  parish 
directors  and  area  consultants?"     The  authors  concluded  that  the 
Nominal  groups  generated  a  hi gher  quality  of  ideas  but  the  Brainstorming 
groups  were  superior  in  terms  of  the  quantity  of  ideas  and  group 
satisfaction.    A  description  of  the  NGT  format  was  not  provided  in 
this  description  of  the  study. 

9.  Nutt  (1976)  compared  the  following  three  processes: 

A.  Systems--first  objectives  were  developed  using  the 
Interactive  method  and  then  the  NGT  was  used  to 
identify  ways  of  providing  services  to  meet  those 
objectives. 

B.  Behavioral --the  NGT  was  used  both  to  identify 
client  problems  and  also  to  identify  solutions  to 
those  problems. 

C.  Heuristic--The  Interactive  technique  was  used  to 
identify  solutions. 
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The  Systems  (combination  Interactive  and  NGT),  the  Behavioral  (NGT) 
and  the  Heuristic  (Interactive)  models  were  compared  in  four  field 
settings.    Performance  was  measured  by  quality,  satisfaction  and 
number  of  new  ideas.    Nutt  concluded  that  the  Behavioral  approach  is 
most  effective  when  new  ideas  (innovation)  are  required  while  the 
Systems  approach  is  best  for  situations  that  demand  high  quality 
decisions.    The  Heuristic  technique  was  found  to  be  ineffective.  The 
author  does  not  provide  a  sufficient  description  of  the  NGT  format  to 
be  able  to  ascertain  if  the  Delbecq  NGT  format  was  followed. 

10.  Burton  et  al .  (1977a)  measured  the  effects  of  group  size 
on  performance  in  Nominal  groups.    Two  hundred  forty-eight  volunteer 
college  students  were  split  into  groups  ranging  from  five  to  seventeen 
members.    Using  a  highly  structured  task,  group  performance  was  compared 
using  three  criteria:    quantity,  quality,  and  perceived  satisfaction. 
They  used  a  (modified)  three  step  NGT  similar  to  that  used  by  Green 
(1975)  (listing,  round  robin,  and  voting).    The  Serial  Discussion  for 
Clarification  (Phase  3)  of  the  Delbecq  et  al .  (1975)  NGT  format  was 
omitted.    The  authors  concluded  that  (1)  the  optimal  NGT  size  is 
"around  10";  (2)  beyond  a  size  of  ten,  the  number  of  unique  ideas 
continues  to  increase  but  group  satisfaction  decreases. 

11.  Whi te  et  al .  (1977)  compared  the  performance  of  NGT,  Brain- 
storming and  Interactive  groups  in  a  field  setting.    Ninety  subjects 
were  divided  into  groups  of  six.    The  task  was  "What  significant 
personal  and/or  organizational  factors  (or  changes)  must  be  taken  into 
account  in  connection  with  a  hospital  expansion  program?"  Performance 
was  measured  by  the  quantity  of  ideas  generated.    They  concluded  that 
there  were  no  significant  differences  in  performance  between  Nominal  and 
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Brainstorming  groups,  but  both  were  superior  to  Interacting  groups. 
The  use  of  a  single  dependent  measure  in  this  study  may  not  provide 
sufficient  data  to  adequately  judge  the  true  performance  of  the 
Nominal  groups  as  the  generation  of  a  large  number  of  ideas  is  not 
the  sole  purpose  of  the  NGT. 

12.    Burton  et  al .  (1977b)  suggest  a  modification  to  the  NGT, 
the  addition  of  the  Gordon  technique.    The  Gordon  technique  is  a 
method  of  hiding  the  real  problem  under  discussion  from  the  participants 
on  the  theory  that  if  they  know  the  actual  problem,  they  will  be  too 
"close  to  it"  to  analyze  it  accurately.    Rather  than  being  told  the  real 
problem  (What  company  problems  limit  organizational  effectiveness?), 
participants  were  asked  to  "list  the  subject  areas  you  would  like  to 
have  included  in  a  follow-on  training  probram."    Subjects  were  thirty 
Management  personnel  in  a  company  Management  Development  program. 
Of  the  thirty-three  ideas  generated,  only  seven  were  perceived  as 
actually  being  "training  problems"  while  the  other  twenty-six  were 
determined  to  be  problems  that  prevented  organizational  effectiveness. 
The  five  step  NGT  procedure  (listing,  round  robin,  voting,  serial 
discussion  and  final  voting)  varied  from  the  Delbecq  et  al .  (1975)  NGT 
procedure  in  that  the  serial  discussion  followed  the  initial  voting 
phase  rather  than  preceding    it.    The  authors  conclude  the  NGT/Gordon 
combination  is  more  effective  than  Brainstorming  or  the  traditional  NGT. 
As  no  control  group  was  provided  for  comparison  purposes,  and  since 
a  modified  NGT  was  utilized,  the  conclusions  of  the  authors  must  be 
seriously  questioned. 


13.    Coo ley  (1977)  conducted  one  of  the  most  ambitious  studies  on 
the  NGT  to  date.    He  attempted  (1)  to  determine  the  optimal  group  size 
(5,  6,  7,  8,  9  or  10  members)  for  NGT  groups,  (2)  to  test  a  modification 
to  the  Delbecq  (1975)  NGT  and  (3)  to  determine  the  appropriate  method 
for  judging  the  quality  of  group  decisions.    Five  hundred  and  forty 
Air  Force  officers  were  used  as  subjects.    The  task  is  one  of  problem 
identification,  using  a  USAF  Base  Resource  Management  Exercise. 

The  N^  Control  group  uses  a  variation  to  the  Delbecq  et  al .  (1975) 
NGT  method.    During  the  round  robin  phase,  (1)  participants  must  read 
their  items  in  the  same  order  they  are  written  on  their  listing  sheet; 
(2)  after  each  item  is  recorded,  the  leader  asks  "Does  anyone  need 
clarification?"    If  so,  only  the  reader  is  allowed  to  clarify;  and  (3) 
after  each  item  is  recorded,  the  leader  asks  "Does  anyone  else  have 
the  item  listed?"    If  so,  the  participant  raises  his/her  hand,  a  check 
is  put  by  the  item  on  the  wall,  and  the  participant  does  not  read 
that  item  when  his/her  turn  comes.    After  the  completion  of  the  round 
robin,  there  is  no  serial  discussion  for  clarification  phase  and  voting 
begins  immediately. 

The       Experimental  group  uses  a  further  modification  of  the  NGT 
format.    After  a  seven  minute  (as  opposed  to  ten  in  N-j )  listing  phase, 
round  robin  recording  begins.    During  this  phase  free  discussion  (as 
opposed  to  only  clarification  in  N^)  is  allowed.    After  this  first 
round  robin,  a  second  three  minute  listing  phase  begins.    Next,  there 
is  a  second  round  robin  phase  with  free  discussion  (as  compared  to  no 
discussion  in  N^).    After  the  second  round  robin,  the  N^  NGT  process 
follows  the  same  procedures  as  the  N-,  groups.    Withholding  was  measured 
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by  comparing  the  individual  listing  sheets  of  each  member  with  the 
ideas  listed  on  the  group's  flip  charts. 

Cooley  concluded  from  his  study  (1)  there  were  no  significant 
differences  in  the  performance  between  N-j  and  N2  NGT  groups.  (2) 
A  NGT  group  size  of  ten  was  the  optimal  group  size.    (3)  "Priority" 
of  items,  as  judged  by  a  panel,  is  the  best  quality  criteria  for  ideas. 
However,  he  further  noted  on  this  priority  rating  that,  "average 
quality  need  not  be  evaluated  for  tasks  similar  to  the  one  used  in  this 
study"  (p.  219).    (4)  Both       and  N2  groups  "withheld"  ideas  from  the 
recorder.    In  fact,  each  of  the  seventy-two  groups  had  ideas  withheld 
and  in  some  as  many  as  33  percent  of  the  listed  ideas  were  withheld 
from  the  group.    Despite  Cooley's  findings  that  the  average  quality  and 
the  quantity  of  ideas  were  closely  correlated,  until  a  definite 
relationship  is  established,  it  would  seem  that  the  continual  use  of 
some  dependent  measure  of  quality  would  be  suggested  as  one  method 
of  judging  the  performance  of  nominal  groups. 


III.    STATEMENT  OF  THE  PROBLEM  AND  OBJECTIVES  OF  THIS  STUDY 

Most  authors  have  hypothesized  that  the  Nominal  Group  Technique 
would  be  more  effective  than  the  traditional  Interacting  Group  process. 
Ten  of  the  thirteen  studies  cited  in  the  previous  chapter  compared  the 
NGT  with  the  Interactive  process.    Of  the  ten  studies,  eight  concluded 
that  the  NGT  was  superior  to  the  Interactive  method.    One  found  partial 
support  for  the  superiority  of  the  NGT,  while  only  one  found  no 
significant  difference  between  the  NGT  and  the  Interactive  process. 

Perhaps  an  even  more  significant  finding  was  the  fact  that  75 
percent  of  the  studies  that  described  their  technique  used  a  Nominal 
group  process  that  varied  significantly  from  the  method  prescribed  by 
Delbecq  et  al.  (1975).    Six  of  the  nine  studies  that  described  the 
technique  used  a  somewhat  modified  NGT  (2,  4,  7,  10,  12,  13).    None  of 
the  six  authors  appeared  to  be  aware  of  the  potential  effects  of  their 
modifications  and  several  stated  they  were  following  the  Delbecq  et 
al.  (1975)  technique  although  the  narrative  descriptions  in  their 
methodology  sections  outlined  a  process  that  varied  significantly  from 
the  published  work  of  Delbecq.    Of  the  thirteen  studies,  only  two 
authors  intentionally  tested  modifications  to  the  Delbecq  technique 
(12  and  13).    One  (13)  utilized  a  control  group  which  did  not  follow 
the  Delbecq  NGT  format  and  the  other  (12)  failed  to  use  a  control  group 
at  all. 

In  summary,  the  review  of  the  literature  leads  to  two  basic 
conclusions.    (1)  The  Nominal  Group  Technique  has  proven  itself  to  be 
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a  more  effective  decision  making  process  than  the  Interactive  method 
for  certain  types  of  tasks.    (2)  A  large  number  of  modifications  to  the 
Delbecq  et  al .  (1975)  N6T  have  been  used  in  research  settings  (whether 
intentionally  or  not)  but  there  have  been  relatively  few  attempts  to 
test  experimentally  whether  these  modifications  increase  or  decrease 
the  effectiveness  of  the  NFT. 

The  Delbecq  NGT  has  now  been  in  use  for  a  full  decade.  Although 
it  has  proven  to  be  a  successful  tool,  it  is  hypothesized  that  further 
refinement  is  possible.    The  primary  question  then  becomes  which  aspects 
are  most  likely  to  lend  themselves  to  improvement? 

It  is  hypothesized  that  one  question  worthy  of  study  is  whether 
inhibitions  about  contributing  ideas  among  participants  can  be  reduced 
or  eliminated  from  the  NGT  process?    This  is  based  on  the  assumption 
that  there  are  inhibiting  factors  working  on  participants  during  an 
NGT  meeting.    Even  though  the  technique  was  primarily  designed  to 
eliminate  inhibiting  factors  that  are  present  during  interactive  group 
meetings. 

The  review  of  the  literature  revealed  only  one  study  that  provided 
evidence  of  inhibiting  factors.    Cooley  (1977)  found  that  every  group 
(a  total  of  seventy-two)  in  his  experimental  design  withheld  ideas 
during  the  NGT  process.    He  further  noted  that  some  groups  withheld  an 
average  of  33  percent  of  their  ideas  from  the  recorder. 

Unfortunately  the  Cooley  study  used  a  modified  version  of  the  NGT 
(see  number  13  in  the  Review  of  Relevant  Empirical  Studies  Chapter) 
which  might  tend  to  increase  inhibitions  beyond  the  level  that  might 
be  found  if  the  traditional  NGT  were  utilized.    More  specifically, 
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Cooley  required  each  participant  to  clarify  his/her  own  ideas.  Such 
a  procedure  encourages  the  linking  of  ideas  with  people  and  if  that 
occurs  individuals  may  be  reluctant  to  present  ideas  that  they  may 
have  difficulty  in  clarifying.    Such  a  procedure  could  drastically 
increase  the  number  of  ideas  that  were  "withheld"  from  the  group. 
If  withholding  could  be  clearly  demonstrated  in  groups  carefully  adhering 
to  the  Delbecq  NGT  format,  this  would  support  the  hypothesis  that  the 
NGT  contains  some  inhibiting  factors.    Thus,  one  variable  considered 
in  this  study  was  the  degree  to  which  NGT  groups  withheld  ideas. 

Possible  Sources  of  Participant  Inhibitions  in  the  Delbecq  NGT 

Some  portions  of  the  NGT  appear,  a  priori,  to  have  the  potential 
for  inhibiting  participants.    Specifically  during  the  round  robin  phase 
(Phase  3)  of  the  Delbecq  et  al .  (1975)  technique  there  are  some 
procedures  that  could  be  inhibiting.    They  include: 

1.    Reading  Aloud--Durinq  the  round  robin  phase  each  parti- 
cipant is  required  to  read  each  of  his/her  ideas  to  the 
recorder.    The  author  hypothesizes  that  by  requiring 
each  individual  to  read  each  of  his  ideas  to  the  group, 
inhibitions  are  raised  unnecessarily.    It  is  suggested 
that  by  requiring  a  shy  or  low  status  person  to  read 
each  idea  he/she  might  be  discouraged  from  submitting 
ideas.    Individuals  who  have  difficulty  reading  aloud 
might  be  reluctant  to  submit  ideas  or  be  encouraged  to 
"pass"  as  an  avoidance  technique  to  escape  the  possibility 
of  stammering,  stuttering  or  mispronouncing  a  word. 
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2.  "Attaching"  an  idea  to  an  indi vidua1--The  NGT  is  de- 
signed to  separate  ideas  from  the  individuals  presenting 
them.    However,  when  a  person  reads  an  idea,  there  is 
always  the  possibility  the  group  will  remember  which 
individual  presented  it.    (This  is  especially  true  for 
the  first  items  on  the  flip  chart  list  as  they  can  easily 
be  linked  to  the  presenter  by  following  the  seating  order 
of  the  participants.)    Thus  a  shy  or  low  status  individual 
might  not  present  an  idea  because  of  this  fear  of  embar- 
bassment.    Similarly,  ideas  that  are  assumed  to  be  of 
general  knowledge  or  that  might  be  near  duplicates  of 
ideas  already  presented  may  be  withheld  for  the  same 
reason.    Ideas  that  are  complex  in  nature  might  also  be 
withheld  by  individuals  who  are  apprehensive  about  possibly 
having  to  explain  them  to  the  group.    Another  possibility 
is  that  controversial  or  negative  ideas  may  be  withheld 
because  of  the  possibility  that  the  "boss"  may  remember 

or  find  out  who  presented  it. 

3.  "Passing"  and  "Nonpassing"  during  the  round  robin  phase-- 
During  the  round  robin  phase  individuals  who  have  no 
ideas  to  present  may  pass  each  round  if  they  wish.  A 
potentially  inhibiting  aspect  of  the  "pass"  procedure 
could  occur  during  the  late  rounds  of  the  round  robin  phase. 
There  might  be  a  tendency  for  premature  closure  in  that 
with  almost  everyone  passing,  an  individual  with  ideas  still 
remaining  might  also  pass  to  avoid  standing  out  or  appearing 
as  a  "know-it  all."    This  fear  of  "standing  out"  by  creative 
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individuals  could  force  the  round  robin  to  end  before 
all  the  possible  ideas  are  presented. 

"Nonpassing"  could  occur  because  individuals  who 
"pass"  during  several  consecutive  rounds  risk  the 
possibility  of  being  identified  as  "noncreati ve. "  This 
fear  could  cause  individuals  to  add  trivial  ideas  to 
avoid  having  to  "pass"  and  thus  slow  up  the  group 
process.    This  fear  could  also  force  participants  to 
present  ideas  that  are  not  completely  thought  out  (or 
ones  that  could  be  condensed)  that  realistically  should 
be  withheld  until  they  are  refined  and  presented  in  a 
later  round. 

4.    CI ari f i cation--After  the  round  robin  phase  is  completed, 
a  serial  classification  phase  begins.    During  this  phase 
of  NGT,  participants  are  allowed  to  present  their  evalua- 
tion of  each  idea  or  request  clarification  of  an  item. 
There  may  be  a  tendency  on  the  part  of  the  recorder 
and/or  the  group  to  ask  the  presenter  of  an  item  to  "clarify" 
his/her  idea.    The  presenter  of  an  item  can  be  identified 
because  the  group  usually  remembers  who  read  the  idea 
aloud  during  the  Round  Robin  phase. 

This  "identification"  is  counter-productive  in  that 
it  forces  an  individual  to  defend  his/her  idea.  The 
NGT  is  designed  to  decrease  this  "identification"  but  in 
practice  it  can  still  occur.    Participants  might  also  be 
discouraged  from  presenting  negative  evaluations  (or 


conversely  be  encouraged  to  give  excessive  positive 
evaluations)  if  the  idea  came  from  a  high  status 
individual. 

The  potential  inhibitions  or  problems  cited  above  could  lead  to 
a  reduction  in  the  effectiveness  of  the  NGT.    Although  there  is  little 
experimental  evidence  to  support  the  existence  of  these  inhibiting 
factors,  casual  field  observations  supported  the  a  priori  hypothesis 
that  they  do  in  fact  exist,  and  that  a  modification  of  the  Delbecq 
NGT  might  lead  to  an  improvement  in  its  effectiveness. 

Mod  NGT 

The  first  objective  of  this  study  was  to  experimentally  test  the 
hypothesis  that  modifications  to  the  silent  writing  and  round  robin 
phases  of  the  Delbecq  NGT  (hereafter  called  Mod  NGT)  would  improve  the 
effectiveness  of  groups  using  the  procedure.    Mod  NGT  alters  the 
Delbecq  NGT  in  the  following  ways: 

1.  During  the  silent  writing  phase  (Phase  1)  each  parti- 
cipant is  given  a  stack  of  blank  3x5  cards  along 
with  a  sheet  describing  the  problem  to  be  considered. 
Each  member  writes  each  of  his/her  ideas  on  a  separate 
3x5  card  (rather  than  on  a  listing  sheet,  which  is 
the  procedure  using  the  traditional  NGT).    The  silent 
writing  phase  lasts  ten  minutes.    All  participants  were 
asked  to  use  pencils  to  ensure  anonymity. 

2.  Mod  NGT  changes  the  Round  Robin  phase  in  that  the 
reading  of  ideas  by  individuals  is  eliminated.  Instead, 
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during  each  round,  each  participant  places  one  3x5 
card  (folded)  in  a  hopper  (hat  or  open  box)  that  is 
passed  around  the  table. 

After  each  person  has  placed  one  card  in  the  hopper, 
the  recorder  reads  each  idea  aloud  and  records  it  on  a 
flip  chart.    By  having  the  idea  written  rather  than  read 
aloud  (as  occurs  using  the  Delbecq  NGT),  there  should  be 
less  likelihood  that  a  recorder  would  be  tempted  to 
record  an  idea  using  his  own  terminology  rather  than  the 
exact  wording  of  the  group  member. 

If  any  ideas  are  unclear  or  are  too  long,  the 
recorder  reads  it  aloud  again  and  asks  that  the  idea  be 
clarified  and/or  shortened  and  resubmitted  during  some 
future  round.    If  during  any  round  a  participant  wishes 
to  "pass,"  he/she  hands  in  a  blank  3x5  card  rather 
than  one  with  an  idea  on  it.    The  round  robin  continues 
until  two  rounds  have  passed  in  which  all  the  cards  in  the 
hopper  were  blank.    All  the  remaining  phases  of  the  NGT 
proceed  according  to  the  Delbecq  technique. 

It  was  hypothesized  that  using  the  3x5  cards,  rather 
than  reading  one's  ideas  aloud,  would  significantly  reduce 
the  potential  inhibiting  factors  that  might  occur  while 
using  the  Delbecq  NGT. 
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NGT  Groups  Using  Sensitive  Vs.  Nonsensitive  Questions 

Delbecq  et  al .  (1975)  assume  that  the  nature  of  the  task  has  no 
effect  on  the  effectiveness  of  the  Nominal  Group  Technique.  However, 
studies  by  Janis  and  Mann  (1976),  Shaw  (1971)  and  Festinger  and  Aronson 
(1960)  have  suggested  that  the  type  of  task  can  affect  the  outcome  of 
a  group  decision  process. 

More  specifically,  the  author  hypothesized  that  the  degree  to 
which  the  NGT  question  might  be  regarded  as  "sensitive"  by  the  group 
members  might  have  some  impact  on  the  NGT  groups  effectiveness.  The 
author  suggests  that  "sensitive"  questions  would  raise  the  inhibitions 
of  the  group  members.    Inhibited  members  in  turn  would  tend  to  be  less 
candid,  with  the  final  result  being  a  less  effective  group. 

Inhibitions  could  be  raised  by  a  sensitive  question  in  Nominal 
Groups  during  the  listing  phase  where  participants  could  fear  their 
ideas  might  be  "traced"  back  to  them  and  thus  their  ideas,  concerns 
and  prejudices  would  be  known  to  all.  Sensitive  questions  could  also 
raise  inhibitions  during  the  discussion  phase  where  one  might  be 
reluctant  to  express  an  opinion  that  might  diverge  from  the  group's 
thinking  because  of  fear  or  just  to  avoid  "making  waves." 

Thus  the  second  hypothesis  considered  in  this  study  was  whether 
the  sensitivity  of  the  NGT  question  has  any  effect  on  the  NGT  group's 
output.    In  order  to  test  this  hypothesis  two  separate  NGT  questions 
were  generated  for  the  study.    The  sensi ti ve  question  related  to  the 
internal  policies  and  procedures  of  the  local  AROTC  unit  thus  raising 
the  possible  inhibiting  factors  of  criticizing  local  policy,  personnel 
and  procedures.    A  second,  nonsensitive  question,  related  to  how  the 
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AROTC  program  could  be  changed  to  make  it  more  attractive  to  potential 
cadets.    This  second  question  related  less  to  those  already  in  the 
program  and  thus  was  designed  to  raise  fewer  inhibitions  than  the 
sensitive  question. 

It  was  suggested  above  that  the  Mod  NGT  would  reduce  any  inhibiting 
factors  that  might  still  be  present  in  the  Delbecq  NGT  process.    If  the 
Mod  NGT  does,  in  fact,  reduce  inhibitions,  one  would  expect  it  to 
perform  much  better  than  the  Delbecq  NGT  (control)  on  sensitive  questions 
when  inhibitions  are  high. 

It  was  also  hypothesized  that  even  generally  nonsensitive  questions 
raise  some  inhibitions  and  as  a  result  the  Mod  NGT  should  still  out- 
perform the  Delbecq  NGT  when  both  used  nonsensitive  questions,  but  per- 
haps not  as  much  as  if  a  sensitive  question  were  used. 

This  study  also  examined  the  question  of  whether  Mod  NGT  groups, 
using  a  sensitive  question,  would  outperform  Mod  NGT  groups  using  a 
nonsensitive  question.    The  author  hypothesized  that  since  the  Mod  NGT 
is  supposed  to  eliminate  most  inhibitions,  it  should  perform  equally  as 
on  sensitive  or  nonsensitive  questions.    Conversely,  the  author  hypothesized 
that  control  groups  using  nonsensitive  questions  would  outperform  control 
groups  using  sensitive  questions  because  fewer  inhibitions  would  be 
raised  by  the  nonsensitive  question. 

One  final  objective  was  to  compare  all  groups  using  a  sensitive 
question  to  all  groups  using  a  nonsensitive  question.    It  was 
hypothesized  that  the  groups  (whether  experimental  or  control)  using 
the  nonsensitive  question  would  face  fewer  inhibitions  and  thus  out- 
perform the  groups  with  the  sensitive  question. 
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The  Withholding  of  Items  from  the  Group 

In  an  earlier  chapter  the  author  noted  that  Cooley  (1977)  found 
that  each  of  his  NGT  groups  (seventy- two)  withheld  ideas  from  the  NGT 
group  to  some  extent.    He  also  noted  that  some  groups  withheld  up  to 
33  percent  of  their  ideas.    Although  that  evidence  must  be  discounted 
due  to  the  fact  that  Cooley  used  a  modified  NGT  process,  the  subject 
of  possible  withholding  was  deemed  worthy  of  inclusion  in  this  study. 

Withholding  occurs  when  a  participant  in  an  NGT  session  writes 
down  an  idea  and  for  some  reason  (possible  inhibitions)  he/she  never 
presents  the  idea  to  the  group.    Withholding  was  measured  by  collecting 
all  3x5  idea  cards  and/or  NGT  question  sheets  from  each  participant 
after  the  NGT  session.    The  items  that  were  withheld  are  determined  by 
comparing  the  individual's  sheet  with  the  items  on  the  group's  list. 
Any  nonpresented  items,  unless  they  were  obvious  duplicates,  were  con- 
sidered to  be  "withheld."    The  objectives  were  (1)  to  determine  if  any 
NGT  groups  withheld  a  significant  number  of  ideas  and  (2)  if  so,  to 
determine  whether  the  experimental  groups  withheld  fewer  ideas  than 
the  Delbecq  NGT  groups. 

In  summary  the  primary  objectives  of  this  exploratory  study  are 
(1)  to  experimentally  test  whether  a  modification  (Mod  NGT)  to  the 
silent  writing  and  to  the  round  robin  phases  of  the  Delbecq  NGT  can 
improve  the  effectiveness  of  the  NGT  by  eliminating  some  of  the  problems 
caused  by  having  individuals  "read"  their  ideas  aloud;  (2)  experimentally 
test  whether  the  relative  sensitivity  of  the  NGT  question  has  any  effect 
on  the  group  effectiveness;  and  (3)  to  measure  to  what  extent  individuals 
withhold  ideas  from  NGT  groups. 
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Although  the  author  hypothesized  that  the  Mod  NGT  and  the  relative 
sensitivity  of  the  NGT  question  would  effect  the  groups'  output,  the 
relative  differences  were  not  expected  to  be  great  due  to  the  fact  that 
the  modification    to  the  NGT  was  not  a  major  change  and  the  relative 
difference  in  degree  of  sensitivity  of  the  two  NGT  questions  was  not 
large.    As  this  study  was  designed  to  be  exploratory,  indications  and 
tendencies  were  considered  to  be  worthwhile  as  well  as  major  statistical 
variations. 

Measure  of  Effectiveness 

The  four  treatments  (independent  variables)  were  applied  to 
Nominal  groups  to  determine  if  they  caused  an  increase  or  decrease  in 
group  effectiveness.    In  this  study  the  measures  of  group  effectiveness 
(dependent  variables)  were  the  level  of  perceived  group  satisfaction, 
the  consensual  judgement  rating  of  the  "top-seven"  items  generated 
by  the  group,  the  quantity  of  ideas  generated  and  the  number  of  items 
withheld  from  the  group.    Each  of  these  dependent  measures  is  discussed 
in  detail  in  Chapter  V. 
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IV.    MAJOR  RESEARCH  HYPOTHESIS 


where  x  =  Mod  NGT 

y  =  Delbecq  NGT 

s  =^  sensitive  NGT  question 

n  =  nonsensitive  NGT  question 

H,       Groups  Nx,  using  Mod  NGT,  will  have  a  higher  level  of  perceived 
group  satisfaction  on  each  of  the  individual  satisfaction 
variables  than  the  Control  groups  Ny. 

H"!       Groups  N^,  using  Mod  NGT,  will  have  an  equal  or  lower  level  of 
0     perceived  group  satisfaction  on  each  of  the  individual  satis- 
faction variables  than  the  Control  groups  Ny. 

Hp      Groups  Nx,  using  Mod  NGT,  will  receive  higher  "top-seven" 
consensual  judgement  ratings  than  the  Control  groups  Ny. 

H„       Groups  Nx,  using  Mod  NGT,  will  receive  the  same  or  lower  "top- 
0     seven"  consensual  judgement  ratings  than  the  Control  groups  Ny, 

H^       Groups  Nx,  using  Mod  NGT,  will  generate  a  greater  quantity 
of  items  than  the  Control  groups  Ny. 

H^      Groups  Nx,  using  Mod  NGT,  will  generate  the  same  or  a  smaller 
0     quantity  of  items  than  the  Control  groups  Ny. 

H^       Groups  Nxs,  using  Mod  NGT  and  a  sensitive  question,  will  have 
a  higher  level  of  perceived  group  satisfaction  on  each  of  the 
individual  satisfaction  variables  than  the  Control  groups  Nyj, 
using  a  sensitive  question. 

H^       Groups  Nxs,  using  Mod  NGT  and  a  sensitive  question,  will  have 

0     an  equal  or  lower  level  of  perceived  group  satisfaction  on  each 

of  the  individual  satisfaction  variables  than  the  Control 

groups  Nys,  using  a  sensitive  question. 

Hg       Groups  Nxs,  using  Mod  NGT  and  a  sensitive  question,  will 

receive  higher  "top-seven"  consensual  judgement  ratings  than 
the  Control  groups  Nyg,  using  a  sensitive  question. 

H^       Groups  Nxs,  using  Mod  NGT  and  a  sensitive  question,  will 
0     receive  the  same  or  lower  "top-seven"  consensual  judgement 

ratings  than  the  Control  groups  Nyg,  using  a  sensitive  question. 
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Hg       Groups  Nxs,  using  Mod  NGT  and  a  sensitive  question,  will  generate 
a  greater  quantity  of  ideas  than  the  Control  groups  Nvs>  using  a 
sensitive  question. 

Hg      Groups  Nxs,  using  Mod  NGT  and  a  sensitive  question,  will  generate 
0     the  same  or  smaller  quantity  of  ideas  than  the  Control  groups  Nys, 
using  a  sensitive  question. 

H-^      Groups  H^j^,  using  Mod  NGT  and  a  nonsensitive  question,  will 

generate  a  higher  level  of  perceived  group  satisfaction  on  each 
of  the  individual  satisfaction  variables  than  the  Control  groups 
Nyp  ,  using  a  nonsensitive  question. 

Groups  Nxn,  using  Mod  NGT  and  a  nonsensitive  question,  will 
0     generate  an  equal  or  lower  level  of  perceived  group  satisfaction 
on  each  of  the  individual  satisfaction  variables  than  the  Control 
groups  Nyn,  using  a  nonsensitive  question. 

Hg      Groups  Nxn,  using  Mod  NGT  and  a  nonsensitive  question  will, 
receive  higher  "top-seven"  consensual  judgement  ratings  than 
the  Control  Groups  Nyn,  using  a  nonsensitive  question. 

Hg      Groups  Nxp,  using  Mod  NGT  and  a  nonsensitive  question  will 
0     receive  tne  same  or  lower  "top-seven"  consensual  judgement 

ratings  than  the  Control  groups  Nyn,  using  a  nonsensitive  question. 

Hg       Groups  Nxn.  using  Mod  NGT  and  a  nonsensitive  question,  will 

generate  a  greater  quantity  of  ideas  than  the  Control  groups  Nyn, 
using  a  nonsensitive  question. 

Hg       Groups  Nxn.  using  Mod  NGT  and  a  nonsensitive  question,  will  generate 
0     the  same  or  a  smaller  quantity  of  ideas  than  the  Control  groups  Nyn, 
using  a  nonsensitive  question. 

H^Q     Groups  Nx,  using  Mod  NGT  will  withhold  significantly  fewer  items 
than  the  Control  groups  Ny. 

H^Q     Groups  Nx,  using  Mod  NGT  will  withhold  an  equal  or  greater 
0    number  of  items  than  the  Control  groups  Nw. 


V.    RESEARCH  DESIGN  AND  METHODOLOGY 


Basic  Design 

This  study  was  a  field  experiment  designed  to  (1)  gather  data  on 
the  effectiveness  of  the  proposed  modification  (Mod  NGT);  (2)  to  deter- 
mine whether  the  relative  sensitivity  of  the  NGT  question  has  any 
effect  on  the  groups'  effectiveness;  and  (3)  to  examine  if  individuals 
in  NGT  sessions  withhold  ideas  from  the  group  and,  if  so,  does  the 
use  of  Mod  NGT  moderate  the  amount  of  withholding. 

The  subjects  for  this  study  were  one  hundred  and  forty-four 
University  of  Florida  Army  ROTC  cadets  who  participated  in  a  semiannual 
Field  Training  Exercise  (FTX).    The  Experimental  Design  includes  24 
groups  divided  into  four  treatments  (X=Mod  NGT,  Y=Delbecq  NGT,  S= 
Sensitive  NGT  Question,  N=nonsensi ti ve  NGT  Question)  as  shown  in  the 
figure.    Subjects  were  randomly  selected  (nonvoluntary)  and  assigned 
to  groups  from  the  cadet  ranks  using  random  numbers  generated  by 
computer.    Each  of  the  basic  groups  (N^^,  N^^,  N^^,  N^^)  had  an  equal 
number  of  subgroups  (six). 

Each  NGT  group  was  comprised  of  seven  members,  six  active  members 
and  a  group  leader  or  recorder.    This  group  size  was  chosen  to  ensure 
as  large  an  N  as  possible  for  statistical  purposes.    A  group  of  seven 
is  within  the  acceptable  range  suggested  by  Delbecq  et  al .  (1975)  of 
five  to  nine.    Considering  the  relatively  small  number  of  groups  using 
each  of  the  four  treatments,  provisions  were  made  to  ensure  stand-by 
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participants  to  ensure  equal  group  sizes  both  within  and  between 
treatments. 

Each  nominal  group  consisted  of  six  active  members  and  a  recorder. 
The  recorder  in  this  study  was  a  "passive"  member  who  was  highly  trained 
to  ensure  that  the  prescribed  NGT  procedure  was  carefully  followed. 
The  use  of  a  "passive"  recorder  in  both  the  Mod  NGT  and  the  Delbecq 
NGT  groups  is  a  variation  from  the  normal  Delbecq  NGT  procedure. 
Under  the  Delbecq  procedure  the  recorder  is  normally  an  active  partici- 
pant and  a  regular  member  of  the  group.    In  this  study,  since  all  groups 
were  ad  hoc  there  were  no  existing  members.    In  addition,  because  it 
was  necessary  to  ensure  for  experimental  purposes  that  the  procedures 
were  strictly  adhered  to,  "passive"  recorders  were  able  to  devote 
more  of  their  time  during  the  NGT  session  to  "policing"  the  group 
than  "active"  recorders  would  have  been  able  to. 

The  recorders  were  senior  AROTC  cadets  who  were  selected  from  a 
larger  group  of  senior  cadets  on  the  basis  of  their  leadership 
abilities.    Each  of  the  recorders  went  through  four  hours  of  training 
on  how  to  conduct  an  NGT  session.    As  part  of  their  training  each  of 
them  participated  in  an  actual  NGT  session.    Recorders  were  also  given 
written  instructions  (see  Appendix  B  &  C)  at  the  NGT  session  and  an 
additional  twenty  minute  refresher  course  in  the  NGT.    As  the  modifica- 
tion to    be  tested  was  an  administrative  type  change  that  did  not 
require  a  behavioral  change  on  the  part  of  the  recorder,  each  recorder 
was  trained  in  both  of  the  NGT  methods  (Delbecq  NGT,  Mod  NGT).  Recorders 
were  randomly  assigned  to  the  groups  by  selecting  a  group  number  (with- 
out any  treatment  condition  identified)  from  a  "hat." 
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The  NGT  Questions 

The  two  NGT  questions  were  derived  in  conjunction  with  the  Army 
ROTC  Cadre  (officers)  and  the  cadet  staff.    The  questions  were  developed 
following  the  four  step  question  development  procedure  outlined  by 
Delbecq  et  al.  (1975,  p.  75).    The  NGT  questions  were  pretested  in  the 
NGT  training  sessions  conducted  for  the  recorders. 

1.  The  "sensitive"  NGT  question  selected  for  use  was: 
"How  can  the  internal  policies,  procedures  or  organiza- 
tion of  our  local  AROTC  program  be  changed  to  make  it 
more  meaningful  to  cadets?"    This  question  was  designed 
to  increase  inhibitions  since  issues  addressed  were 
local  issues  that  directly  affected  the  daily  lives  of 
the  cadets  who  were  already  in  the  program.    The  question 
was  pretested  among  the  senior  cadet  officers  to  ensure 
that  they  became  emotionally  involved  with  the  issues 
raised  by  the  question.    As  the  items  generated  in  the 
pretest  included  questions  about  the  competence  of  both 
cadet  and  cadre  officers,  the  reasonableness  of  academic 
and  haircut  policies  and  other  emotional  issues  on  the 
local  level,  the  question  was  judged  as  acceptable. 

2.  The  "nonsensi ti ve"  NGT  question  selected  for  use  was: 
"Given  existing  monetary  and  personnel  limitations,  are 
there  any  ways  of  making  AROTC  more  attractive  to 
potential  recruits?"    This  question  v/as  designed  to  be 
more  neutral  or  less  inhibiting  than  the  "sensitive" 
question.    This  question  addressed  issues  that  were  less 
relevant  to  the  cadets  already  enrolled  in  the  ROTC  program. 
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The  responses  generated  during  the  pretest  touched  on 
less  emotional  issues  such  as  the  number  of  public* 
demonstrations,  better  advertising,  and  visiting  more 
high  schools.    Since  the  items  generated  by  the  non- 
sensitive  question  seldom  mentioned  personalities  and 
generally  centered  on  the  activities  of  the  cadre 
rather  than  the  cadets,  the  nonsensitive  question  was 
judged  to  be  relatively  less  inhibiting  than  the 
sensitive  question. 
The  two  questions  selected  for  use  are  not  radically  different  in 
regard  to  degree  of  sensitivity.    This  is  due  to  the  fact  that  (1)  ' 
AROTC  officials  would  not  allow  extremely  sensitive  questions  to  be 
used  and  (2)  the  use  of  any  totally  nonsensitive  question  would  lead 
to  a  low  level  of  involvement  (or  boredom)  on  the  part  of  the  cadets. 

Methods  and  Procedures 

The  experiment  was  conducted  according  to  the  following  format: 

1.  Arrangements  were  made  and  final  approvals  were 
obtained  from  the  administration  of  the  Army  ROTC 
program. 

2.  The  AROTC  program  was  asked  to  develop  and  approve 
the  problem  statement  according  to  the  four  step 
Delbecq  process. 

3.  The  FTX  site  (Oleno  State  Part,  High  Springs,  Florida) 
was  inspected  to  ensure  there  was  adequate  indoor  space 
for  each  of  the  twenty-four  NGT  sessions. 
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4.  Recorders  were  trained  for  two  hours  on  both  NGT 
procedures  two  weeks  prior  to  the  FTX.    Possible  viola- 
tions of  the  procedures  were  demonstrated  and  ways  of 
handling  them  were  discussed  in  detail.    Each  recorder 
was  given  detailed  instructions  on  how  each  NGT  session 
was  to  be  constructed. 

5.  Recorders  participated  in  an  actual  two  hour  NGT  session, 
one  week  prior  to  the  FTX.    This  was  to  ensure  each  had 
some  "hands  on"  experience  with  the  process. 

f).    The  NGT  questions  and  the  satisfaction  questionnaire 
were  pretested  at  the  recorders  NGT  training  sessions. 

7.  The  responses  generated  by  the  recorders  NGT  sessions 
were  presented  to  the  Cadre  to  ensure  the  responses 
were  in  line  with  their  expectations.    The  nonsensitive 
NGT  question  was  modified  slightly  as  a  result  of  this 
step.    The  final  NGT  questions  are  listed  in  the  previous 
section. 

8.  Satisfaction  scores  from  the  recorder  sessions  were  factor 
analyzed  and  the  instrument  was  refined  to  eliminate  some 
ambiguous  phrases. 

9.  Tape  recorders  were  acquired  to  ensure  at  least  a 
random  sampling  of  the  NGT  sessions.    The  sessions  were 
recorded  to  allow  for  a  later  check  on  adherence  to 
procedures. 

10.    The  mess  and  the  recreation  hall  were  prepared  for  the  NGT 
sessions  on  the  day  of  the  FTX  (October  27,  1978). 
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11.  Recorders  were  randomly  assigned  to  groups  and  a 
final  twenty  minute  training  session  was  held. 

12.  Each  recorder  was  given  the  necessary  materials  prior 
to  the  NGT  session.    Materials  included  either  a 
detailed  instruction  sheet  for  the  N    groups  (see 

A 

Appendix  B)  or  an  instruction  sheet  for  the  groups 
(see  Appendix  C),  seven  sheets  containing  the  NGT 
question  (either  sensitive  or  nonsensiti ve) ,  four  flip 
chart  sheets,  a  tape  recorder  (with  the  exception  of 
groups  17-20),  a  marking  pen,  seven  pencils,  masking 
tape,  twelve  group  members  individual  voting  sheets, 
two  group  leaders  summary  voting  sheets,  six  satisfac- 
tion questionnaires  (see  Appendix  D) ,  and  two  blank 
90  minute  cassettes.    In  addition,  N    group  leaders 

X 

received  approximately  ninety  blank  3x5  cards  for 
presenting  ideas. 

13.  The  cadets  were  called  to  formation  and  randomly 
assigned  to  groups.    The  group  leaders  took  their  six 
members  to  their  assigned  table  and  the  session  began. 

The  Ny  Control  Group  NGT  Sessions 

The  control  group  sessions  used  the  Delbecq  NGT  format.  The 
session  began  by  counting  the  members  to  ensure  the  proper  number  of 
participants  (6).    The  recorder  began  the  NGT  session  by  describing 
objectives,  procedures  and  rules  for  the  NGT  session.    This  introduction 
lasted  no  longer  than  five  minutes.    The  recorder  then  noted  the  time, 
started  the  tape  recorder  and  the  session  began.    The  recorder  passed 
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out  the  NGT  question  and  the  listing  sheets  to  each  participant. 
The  problem  statement  was  read  and  the  ten  minute  silent  writing 
period  began.    The  session  proceeded  through  the  round  robin,  serial 
discussion  and  voting  phases  as  described  by  Delbecq  et  al.  (1975). 
During  the  voting  phase  the  group  was  asked  to  select  the  "top-seven" 
ideas  by  ranking  (using  7  for  the  best  and  1  for  their  last  selection 
for  the  "top-seven").    As  the  session  concluded,  the  recorder  noted 
the  time  and  turned  off  the  tape  recorder. 

The  participants  were  asked  to  provide,  if  they  chose  to,  any 
clarifying  statements  that  might  make  their  "top-seven"  items  easier 
to  understand  by  outsiders  who  were  not  exposed  to  the  group's  discus- 
sion.   Brevity  was  encouraged  and  a  short  paragraph  on  each  item 
was  the  outside  limit  on  comments.    These  comments  were  used  by  the 
judges  to  aid  them  in  reaching  a  consensual  judgement  rating  of  each 
group's  output.    The  recorder  then  handed  out  the  satisfaction  ques- 
tionnaires; after  they  were  completed  he  collected  them  and  the  listing 
sheets  used  by  each  of  the  participants.    The  cadets  were  thanked  for 
their  cooperation  and  were  asked  to  leave  quietly.    The  recorder  then 
filled  out  the  summary  voting  form  noting  the  elapsed  time,  the  "top- 
seven"  ideas  (with  clarifying  statements),  the  total  number  of  ideas 
listed  on  the  flip  chart  sheets  and  any  problems  or  variations  to  the 
prescribed  format  that  might  have  occurred  (and  during  what  portion  of 
the  session  it  occurred).    The  recorder  noted  the  group  number  on  each 
of  the  flip  chart  sheets,  the  listing  sheets  collected  from  the 
participants,  the  two  cassette  tapes  and  each  of  the  satisfaction 
questionnaires  filled  out  by  the  participants.    All  of  the  materials  were 
then  placed  in  a  large  envelope  and  turned  into  one  of  the  four  super- 
visors who  were  monitoring  the  NGT  sessions. 
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The  Mod  M6T  Session 

The       Experimental  groups  used  the  Mod  NGT  procedure  in  their 

sessions.    The  N    groups  proceeded  exactly  as  the  N    groups  described 
X  y 

above  with  the  following  exceptions: 

1.  During  the  silent  writing  phase  each  participant  listed 
each  idea  on  a  separate  3x5  card  rather  than  on  a 
listing  sheet. 

2.  During  the  round  robin  phase  each  participant  dropped 
one  folded  3x5  card  in  the  hopper  each  round.  The 
card  could  either  contain  an  idea  or  was  blank  if  the 
individual  wanted  to  "pass."    The  recorder  read  each 
idea  aloud,  and  listed  it  on  the  flip  chart.    Any  idea 
that  was  unclear  to  the  recorder  or  too  lengthy  was  not 
listed.    Instead  the  recorder  read  it  again  out  loud 

and  suggested  to  the  group  that  the  idea  needed  rewording 
(or  shortening),  and  that  the  idea  should  be  resubmitted 
by  someone  in  a  subsequent  round. 

3.  After  the  N    session  concluded,  the  recorder  asked  each 

X 

participant  to  drop  all  his  remaining  3x5  cards  in  the 
hopper  (whether  used  or  blank).    The  recorder  stressed 
to  the  group  that  cards  that  were  not  submitted  during 
the  session  must  be  put  in  the  hopper  regardless  of  why 
the  participant  never  presented  the  idea  to  the  group. 
After  the  group  left,  the  recorder  was  asked  to  sort  out 
any  cards  containing  unsubmitted  ideas  and  turn  them  into 
the  project  leader  along  with  all  the  cards  submitted  by 
the  group. 
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Other  Procedures 

A.  To  ensure  that  experimental  procedures  were  followed, 
four  supervisors  were  present  to  answer  questions  and 
spot  potential  problems. 

B.  Sensitive  and  nonsensitive  NGT  questions  were  alternated 
to  eliminate  any  copying  of  ideas  from  adjacent  groups. 

C.  Cadre  officers  were  not  allowed  to  directly  oversee  any 
group  so  that  the  participants  felt  free  to  express 
their  ideas. 

Problems 

The  experiment  went  very  smoothly  with  the  following  exceptions: 

1.  One  hall  became  somewhat  stuffy  and  noise  from  other 
tables  was  noticeable. 

2.  Several  tape  recorders  malfunctioned  causing  several 
groups  to  go  unrecorded. 

3.  The  presence  of  the  tape  recorders  was  obviously  a 
distraction  to  the  group  leader  and  it  was  received 
apprehensively  by  some  participants. 

Measurement  of  Variables  and  Data  Collection 

The  dependent  variables  that  were  selected  to  test  the  relative 
effectiveness  of  the  two  Nominal  Group  Techniques,  and  the  part 
played  by  sensitive  and  nonsensitive  questions  were: 

1.    The  perceived  group  satisfaction  with  each  of  the 
sixteen  individual  satisfaction  variables; 
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2.  The  consensual  judgement  rating  of  the  "top-seven" 
items  selected  by  the  group; 

3.  The  quantity  of  ideas  generated  by  the  group;  and 

4.  Whether  individuals  withheld  items  from  the  NGT  group. 
The  rationale  for  selecting  each  of  these  measures  and  a  discussion 
of  the  data  collection  procedures  is  presented  in  the  following  sec- 
tions. 

Level  of  Perceived  Group  Satisfaction 

Group  satisfaction  was  chosen  as  a  dependent  measure  because  even 
when  highly  creative  decisions  are  developed,  if  the  decision  makers 
feel  dissatisfied  with  the  process  or  cannot  accept  the  solution, 
the  decision  may  fail  to  be  adopted  (Van  de  Ven,  1974).  More 
specifically,  since  in  this  study  it  was  hypothesized  that  the  proposed 
modifications  will  make  the  NGT  process  more  appealing  to  the  partici- 
pants, there  must  be  some  objective  measure  to  ensure  the  modifications 
increase  the  groups  satisfaction  with  the  technique. 

There  are  sixteen  independent  measures  of  perceived  group  satis- 
faction in  this  study.    Eleven  of  them  cover  satisfaction  with  the 
group  process  while  five  cover  satisfaction  with  the  groups  output. 
Participant  satisfaction  was  measured  using  a  two  part  satisfaction 
questionnaire. 

The  questionnaire  (see  Appendix  D)  was  filled  out  by  each 
participant  immediately  following  the  conclusion  of  their  NGT  session. 
Although  parts  of  the  questionnaire  were  derived  from  instruments 
utilized  by  Van  de  Ven  (1974),  Frederick  (1976)  and  Cooley  (1977), 
many  of  the  questions  were  developed  to  fit  the  specific  needs  of  this 
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study.    The  questionnaire  was  pretested  for  clarity  using  NGT 
recorders  and  other  AROTC  personnel  as  subjects.    Each  of  the  sixteen 
satisfaction  questions  is  answered  on  a  0-10  Likert-type  scale. 

Each  individual  satisfaction  variable  was  treated  separately 
as  an  individual  measure. 

Process 

1.  Traced  Back--can  an  idea  you  presented  to  the 
group  be  traced  back  to  you? 

2.  Refuse  to  Defend--how  free  were  you  to  refuse 
to  defend  your  ideas? 

3.  Free  to  Pass— did  you  feel  free  to  pass  without 
embarrassment? 

4.  Disagree  with  an  Item--were  you  free  to  disagree 
with  an  item? 

5.  Seek  Clarification--were  you  free  to  seek 
clarification  of  items? 

6.  Fair  Consideration--were  your  ideas  given  fair 
consideration  by  the  group? 

7.  Opportunity  to  Present--did  you  have  enough 
opportunities  to  present  your  ideas? 

8.  Time  Efficient--was  the  process  time  efficient? 

9.  Not  Dominated--was  the  group  dominated  by  anyone? 

10.  Group  Leader--satisfied  with  the  performance  of  the 
group  leader? 

11.  All  Aspects--satisfied  with  all  procedural  aspects 
of  the  process? 

Output 

12.  Quantity- -satisfied  with  the  quantity    of  ideas? 

13.  Qual ity--satisfied  with  the  quality  of  ideas? 

14.  Fair  Representation--satisfied  items  generated  are 
a  fair  representation  of  the  groups'  feeling? 
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15.  Output  as  your  Position--satisf1ed  to  present 
groups  output  as  your  position? 

16.  Overall  Goals--clid  the  group  attain  its  overall 
goals? 

Consensual  Judgement  Rating  of  the  Ideas  Generated 

Cooley  (1977)  conducted  a  comprehensive  review  of  the  literature 
related  to  measuring  the  quality  of  group  decisions.    He  notes  that 
although  there  appears  to  be  a  consensus  among  researchers  that  the 
quality  of  the  idea  or  solution  is  a  critical  element  in  the  measure- 
ment of  effectiveness,  some  have  evaded  the  issue  by  noting  that  an 
objective  measure  of  quality  may  not  exist.    His  review  also  indicated 
that  no  standardized  procedure  exists  so  one  must  develop  an  approach 
for  evaluating  quality  that  meets  one's  specific  needs. 

Delbecq  and  Van  de  Ven  (the  developers  of  the  NGT)  do  not  utilize 
a  quality  measure  in  their  experimental  testing  of  the  NGT  (Van  de  Ven 
and  Delbecq,  1974).    Cooley  (1977)  attempts  to  develop  a  suitable 
quality  standard    for  the  NGT  but  concludes  that  measures  of  quality 
are  subjective  in  nature  and  further  suggests  that  "average  quality 
need  not  be  evaluated  for  tasks  similar  to  the  one  used  in  this 
study"  (p.  219).    He  also  concluded  that  "Quantity  may  in  fact  be  the 
only  measure  of  effectiveness  required"  (p.  219). 

Despite  the  problems  involved  in  measuring  quality,  one  measure, 
the  quality  of  the  "top-items"  generated  by  the  group,  used  by  Cooley 
(1977)  seems  most  appropriate  for  Nominal  groups.    Since  Nominal  groups 
differ  from  other  group  decision  techniques,  in  that  the  group  must 
evaluate  and  select  the  highest  priority  items,  a  quality  measure  of 
this  evaluation  and  selection  process  would  be  an  important  indicator 
of  the  effectiveness  of  the  Nominal  group.    The  term  quality  is  somewhat 
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misleading  in  that  there  is  no  absolute  measure  of  the  quality  of 
an  idea.    The  closest  alternative  has  been  having  a  panel  of  experts 
make  ratings  or  judgements  as  to  the  relative  importance  or  quality 
of  an  item.    In  this  study  a  facsimile  of  a  quality  rating  is  utilized 
as  the  second  dependent  measure.    It  is  called  the  Consensual  Judgement 
Rating.    Thus  the  second  dependent  measure  utilized  in  this  study  is 
the  Consensual  Judgement  Rating  of  the  "top-seven"  items  generated  by 
each  group. 

The  "top-seven"  items  selected  by  each  group  were  noted  by  the 
recorder  at  the  end  of  each  session.    The  judging  procedures  used  in 
this  studj^are  partially  adopted  from  those  developed  by  Cooley  (1977) 
in  his  study  of  group  effectiveness  using  a  modified  NGT.    Six  judges 
were  used  in  the  judging  process.    Three  cadre  officers  and  three 
cadet  leaders  were  selected  who  were  familiar  with  AROTC  program. 

Each  of  the  six  judges  was  given  a  rater's  instruction  sheet 
(see  Appendix  E)  and  two  stacks  of  eighty-four  responses.    The  first 
stack  contained  responses  from  question  A  (the  nonsensitive  question) 
and  the  second  contained  the  responses  to  question  B  (sensitive).  Any 
clarifying  comments  included  by  the  group  were  also  placed  on  the  cards. 
As  twelve  groups  answered  each  question  and  each  group  generated  seven 
items,  the  total  for  each  was  eighty- four.    Each  response  was  numbered 
but  they  could  not  in  any  way  be  traced  to  the  group  or  process  it 
originated  from.    Each  rater  was  asked  to  rate  each  item  as  it 
responded  to  the  NGT  question  on  feasibility,  importance  and/or  merit. 
There  were  no  exact  duplicates  and  judges  were  asked  to  clip  "duplicates" 
together  so  they  would  be  judged  as  equal.    Each  rater  used  a  modified 
alternation  ranking  procedure  which  placed  equal  (items  of  similar 


42 


quality)  items  in  the  same  category  starting  at  the  very  best,  then 
alternating  to  the  very  worst;  then  grouping  the  second  best  and  the 
second  worst  etc.  until  all  items  were  ranked.    The  same  procedure 
was  then  followed  for  question  B. 

Interater  reliabilities  were  then  run  using  Kendall's  Coefficient 
of  Concordance  "W"  (Siegel,  1956).    Kendall's  "W"  is  a  nonparametri c 
correlation  coefficient  which  ranks  judges  ratings  on  an  ordinal  . 
scale.    Interater  reliabilities  were  run  between  cadre  officers, 
between  cadet  raters,  and  between  all  six  combined  using  the  BMDP  (P3S) 
statistical  package  (Brown,  1977).    The  Kendall's  "W"  correlations  for 
each  question  were  (1)  Cadre  A  =  .60,  B  =  .64;  (2)  Cadets  A  =  .57, 
B  =  .47;  and  (3)  Cadre  and  Cadets  combined  A  =  .39  and  B  =  .35.  All 
coefficients  were  significant  to  the  .05  level.    The  coefficients  for 
the  cadre  were  moderately  high  indicating  a  reasonable  amount  of  agree- 
ment between  the  cadre  officers.    The  coefficients  for  the  cadets 
demonstrate  that  there  was  only  a  moderate  amount  of  agreement  between 
the  cadet  officers  on  the  relative  quality  of  the  items  generated. 
The  coefficients  for  the  combined  (cadre  and  cadets)  ratings  indicate 
a  relatively  low  level  of  agreement  between  the  cadre  and  the  cadet 
officers.    Such  a  "low"  coefficient  was  also  an  indication  that  the 
consensual  judgement  scores  of  the  cadre  and  the  cadets  might  tend  to 
move  in  opposite  directions. 

Quantity  of  Ideas 

Van  de  Ven  (1974)  notes  that  the  crucial  factor  facilitating 
creative  decision  making  (during  the  fact  finding  stage  of  problem 
solving)  is  the  generation  of  a  large  number  of  ideas  relevant  to  the 
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problem.    During  the  next  step  of  creative  decision  making  (evaluation) 
a  large  number  of  ideas  provides  the  opportunity  for  the  group  to  under- 
take an  exhaustive  evaluation  of  the  problem  area  and  a  reconceptual iza- 
tion  of  solutions  which  have  not  previously  existed.    The  quantity  of 
ideas  was  selected  as  a  dependent  measure  because  "the  greater  the 
quantity  of  ideas  identified  by  a  decision  process,  the  greater  number 
of  ideas  considered  for  evaluation,  and  the  greater  the  potential  for 
creative  decision  making"  (Van  de  Ven,  1974,  p.  26).    A  review  of  the 
literature  demonstrated  that  nearly  every  major  study  on  the  NGT  has 
included  a  quantity  variable  as  one  of  its  dependent  measures. 

In  this  study  the  quantity  of  ideas  was  obtained  from  the  flip 
chart  recording  sheets  utilized  in  each  group.  The  quantity  measure 
is  a  numerical  score  obtained  by  simply  counting  the  number  of  ideas 
generated  by  the  group.  Although  unrecorded  items  (those  items  listed 
by  individuals  but  not  presented  to  the  group)  will  be  considered  in 
this  study,  they  are  not  considered  as  part  of  the  quantity  of  items 
generated  by  the  group. 

Unrecorded  Items 

Listing  sheets  (and  the  3  x  5  cards  in  the  N    groups)  were  col- 

X 

lected  from  each  participant  following  the  NGT  session.    The  group 
number  was  placed  on  each  sheet  and  the  project  leader  later  compared 
ideas  listed  by  each  participant  to  the  ideas  recorded  on  the  flip 
charts  to  see  if  there  was  any  evidence  of  the  withholding  of  ideas. 
An  unrecorded  item  that  was  a  close  duplicate  of  a  recorded  item  was 
not  counted  as  unrecorded. 
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Additional  Data  that  Were  Collected 

.  In  addition  to  the  primary  measurements  (satisfaction,  consensual 
judgement  rating,  quantity  and  unrecorded  ideas)  other  data  were  col- 
lected for  comparison  purposes.    These  additional  measurements  include 
■•    1.    Time--The  elapsed  time  of  each  NGT  session  was  noted 

i-  ■ 

by  the  recorder.    This  might  provide  evidence  as  to 
whether  any  of  the  treatments  took  significantly  less 
or  more  time  to  complete. 

2.  Subjective  Eval uations--As  part  of  the  satisfaction 
questionnaire,  each  participant  was  asked  for  narrative 
comments  relating  to:    What  aspects  of  the  process  did 
you  like  most?    what  aspects  of  the  process  did  you  like 
the  least?;  do  you  have  any  suggestions  for  the  improve- 
ment of  the  process?    and  do  you  have  any  general  com- 
ments about  the  process,  the  recorder  or  the  experiment? 

3.  Adherence  to  Experimental  Procedures--Twenty  NGT  groups 
were  given  tape  recorders  so  that  a  permanent  record  would 
be  available  for  use  by  the  project  leader  in  determining 
whether  the  proper  experimental  procedures  were  followed. 
Participants  were  assured  that  safeguards  were  instituted 
to  ensure  the  tapes  would  only  be  used  by  the  experimenter 
and  would  not  be  available  to  superiors.    Of  those 
original  twenty,  seven  recorders  malfunctioned,  leaving 
thirteen  groups  that  were  at  least  partially  recorded. 
Although  background  noise  is  bad,  review  of  the  tapes  is 
still  possible. 
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The  project  leader  sampled  major  portions  of 
each  tape  to  ensure  the  NGT  process  was  followed.  This 
information  was  used  in  conjunction  with  recorder  com- 
ments; observations  of  the  four  supervisors  present  at 
the  NGT  sessions;  and  evidence  obtained  by  examining 
the  listing  sheets,  recording  sheets  and  voting  sheets 
to  determine  if  any  group  violated  experimental 
procedures.    After  close  examination,  two  groups  were 
found  to  be  questionable  in  some  aspects  but  acceptable 
overall.    In  the  first  group,  on  two  occasions,  the 
leader  asked  that  an  item  be  partially  explained  before 
he  wrote  it  on  the  flip  chart.    In  the  second  group 
the  leader  laughed  when  a  participant  presented  one 
particular  idea. 

4.  Recorder  Comments--At  the  conclusion  of  each  session 
the  recorder  was  asked  to  report  his  comments  on  how 

the  meeting  progressed  and  whether  any  problems  occurred. 
If  problems  occurred,  the  recorder  was  asked  to  note  at 
which  point(s)  in  the  meeting  did  the  problem(s)  occur. 

5.  Other  Data  Were  Col  1  ected  on: 

A.  the  sex  of  the  group  leader; 

B.  the  number  of  items  crossed  out  by  each  group; 

C.  the  number  of  votes  taken  by  the  group;  and 

D.  the  number  of  votes  the  group  gave  to  each  of  the 
top  seven  items. 
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Statistical  Analysis  of  Data 

The  data  gathered  from  groups  Nxs,  Nxn^  Nyg,  and  Nyp  were  used  to 
test  the  research  hypotheses  presented  in  Section  IV.    The  experiment 
was  a  completely  randomized,  fixed  effects,  post-test  only  control 
group  design.    The  two  primary  independent  variables  were  Mod  NGT  and 
the  sensitive  and  nonsensitive  question.    The  primary  dependent  measures 
were  the  perceived  group  satisfaction,  the  consensual  judgement  rating 
of  the  "top-seven,"  the  quantity  of  ideas  and  the  number  of  items  with- 
held. 

Data  generated  by  each  group  were  coded  and  keypunched.    Data  were 
analyzed  using  the  BMDP  p  series  statistical  package.    In  addition 
histograms  and  a  variety  of  descriptive  statistics  were  run  for  each 
of  the  major  variables.    The  data  from  the  dependent  variables  were 
treated  in  the  following  manner. 

Perceived  Group  Satisfaction 

Each  member  of  every  subgroup  completed  a  two  part  satisfaction 
questionnaire.    Each  of  the  sixteen  items  was  scored  on  a  0-10  scale 
by  each  participant.    As  there  is  no  evidence  to  conclude  that  the 
interval  between  0  and  1,  1  and  2,  9  and  10  were  equivalent  on  an 
interval  scale,  the  satisfaction  data  were  treated  as  ordinal.  Com- 
parisons were  made  between  group  means  using  the  nonparametric  Kruskal- 
Wallis  univariate  ANOVA  test.    A  Mann-Whitney  "U"  statistic  was  also 
generated  for  each  satisfaction  variable.    The  Null  Hypothesis  for 
each  comparison  was  that  each  of  the  treatments  have  equal  means  on 
the  dependent  variable  satisfaction.    Each  of  the  sixteen  individual 
satisfaction  variables  were  treated  independently  and  analyzed  for 
each  of  the  following  treatments: 
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X  where  x  =  Mod  NGT 

X  y  -  Delbecq  NGT 

^<;n  ^  ^Kn  ^  ^  Sensitive  NGT  question 

oil  y 

N     X  N  n  =  nonsensitive  NGT  question 

AO  All 

An  significance  levels  were  set  at  .05  for  two  tailed  distributions, 


Consensual  Judgement  Rating 

Each  of  the  subgroups  selected  (voted  on)  its  "top-seven"  items. 
These  items  were  compiled  and  rated  by  judges  on  a  thirteen  item 
(maximum)  alternation  ranking  scale.    As  the  judges  were  asked  to  rank 
(rather  than  rate)  each  item,  using  a  nonprescri bed  number  of  ranks, 
the  ratings  were  treated  on  an  ordinal  scale.    Comparisons  were  made 
between  group  means  using  the  nonparametric  Kruskal-Wallis  univariate 
ANOVA  test.    A  Mann-Whitney  "U"  statistic  was  also  generated  for  each 
comparison  between  groups.    The  Null  Hypothesis  for  each  comparison 
was  that  each  of  the  treatments  had  equal  means  on  the  dependent 
variable  consensual  judgement  rating.    Consensual  judgement  scores  were 
compared  between  each  of  the  following  treatments: 
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All  significance  levels  were  set  at  .05  for  two  tailed  distributions. 

Quantity  of  Ideas 

The  total  number  of  ideas  generated  by  each  subgroup  will  be 
treated  as  that  subgroup's  quantity  score.    The  quantity  score  was 
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treated  as  an  interval  scale.    Univariate  ANOVA  was  used  to  test  the 
Null  Hypothesis  that  the  treatments  had  equal  means  on  the  dependent 
variable  quantity  of  ideas.    A  students  "t"  statistic  was  also 
generated  for  each  comparison.    The  quantity    of  ideas  were  compared 
between  each  of  the  following  treatments: 
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All  significance  levels  were  set  at  .05  for  two  tailed  distributions. 

Number  of  Withheld  Items 

At  the  conclusion  of  each  session  3x5  cards,  listing  sheets 
and  flip  chart  pages  were  collected  by  the  group  leader.  Later 
comparisons  were  made  between  listed  ideas  and  recorded  ideas  to  see 
which  were  withheld  from  the  group.    The  number  of  withheld  ideas 
was  treated  on  an  interval  scale.    Univariate  ANOVA  was  used  to  test 
the  Null  Hypothesis  that  the  treatments  had  equal  means  on  the 
dependent  variable  number  of  withheld  items.    A    student  "t"  statistic 
was  also  generated  for  each  comparison.    The  number  of  withheld  items 
were  compared  between  each  of  the  following  treatments: 
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All  significance  levels  were  set  at  .05  for  tm  tailed  distributions, 
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Operational  Definitions 

FoV  purposes  of  this  study  the  terms  in  this  section  shall  be 
defined  as  noted: 
NGT--The  Nominal  Group  Technique. 


Delbecq  NGT--The  particular  variation  of  the  NGT  described  by  Delbecq 
et  al.,  (1975)  on  pages  56-69  (A. K. A. —Delbecq  technique  or  the 
traditional  NGT). 

Mod  NGT--The  particular  variation  of  the  Delbecq  NGT  which  requires 
participants  to  place  their  ideas  on  3  x  5  cards  rather  than  reading 
them  aloud  to  the  group.    Mod  NGT  is  described  in  detail  in  Chapter  III. 

Sensitive  NGT  Question--The  NGT  question  relating  to  the  improvement 
of  the  local  ROTC  program.    Designed  to  raise  the  inhibitions  of 
participants  to  a  level  higher  than  if  a  nonsensitive  question  were 
used  (see  Chapter  III). 

Nonsensitive  NGT  Question— the  NGT  question  relating  to  making  the 
ROTC  program  more  attractive  to  potential  recruits.    Designed  to 
raise  a  lower  level  of  inhibitions  than  the  sensitive  question  (see 
Chapter  III). 

Interactive  Group--The  traditional  format  for  most  meetings  that  has 
little  structure  and  few  controls.    There  is  (1)  unstructured  group 
discussion  for  obtaining  and  pooling  ideas  of  the  participants  and 
(2)  majority  voting  in  priorities  by  hand  count  (Van  de  Ven,  1974). 

Quantity  of  Ideas  Genera ted--A  dependent  variable  consisting  of  the 
number  of  ideas  generated  by  the  group.    The  quantity  measured  is 
described  in  detail  in  Chapter  V. 

Consensual  Judgement  Rating  of  Ideas  Generated--For  this  study,  the 
consensual  judgement  rating  is  the  rating  assigned  to  each  item  by  a 
panel  of  independent  judges  using  a  modified  alternation  ranking 
(see  Chapter  V) . 

Perceived  Group  Satisfaction--The  rating  of  each  participant  on  a  0-10 
Likert-type  scale  for  each  of  sixteen  independent  satisfaction 
variables  (see  Chapter  V). 

Improved  Ef fecti veness--For  this  study,  when  the  measured  level  of 
perceived  satisfaction,  consensual  judgement  ratings,  quantity  of 
ideas  or  the  number  of  withheld  ideas  increases  so  that  the  new  level 
is  significantly  (statistically  at  the  .05  level)  higher  than  that  of 
the  control  group. 
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Elapsed  Time  of  the  NGT— The  recorded  clock  time  from  the  reading  of 
the  Problem  Statement  to  the  end  of  the  final  voting  phase. 

Undisclosed  Items--For  this  study  any  item  written  down  by  a 
participant  on  his/her  listing  sheet  (or  3  x  5  cards)  but  not  presented 
to  the  group  for  recording,  regardless  of  the  reason  (close  duplicates 
excl uded) . 

Ny— The  set  of  Control  Groups  that  used  the  Delbecq  NGT. 

ITx— The  set  of  Experimental  Groups  that  used  the  Mod  NGT  variation  of 
the  NGT. 

N^-'The  set  of  groups  using  the  sensitive  NGT  question. 

N^--The  set  of  groups  using  the  nonsensitive  NGT  question. 

Recorder^-- For  this  study  a  passive  participant  in  an  NGT  session  who 
explains  procedures,  keeps  order,  records  ideas  on  flip  charts,  and 
counts  the  votes  during  the  voting  phase.    He/she  does  not  add  ideas, 
clarify  ideas  or  participate  in  the  voting  of  the  group. 


VII.    RESEARCH  RESULTS 


All  dependent  measures  (Satisfaction,  Consensual  Judgement, 

Quantity,  Items  Withheld)  were  compared  across  each  of  the  following 

treatments  using  the  data  generated  by  this  experiment: 

with  N 
■  X  y 

.    -.  N  ^    with  N 

xs  ys 

N  „    with  N 
xn  yn 

N  ^    with  N 
xs  xn 

N       with  N 

ys  yn  ■ 

with  N 
s  n 

where    x  =  Mod  NGT  (Exper) 

y  =  Delbecq  NGT  (control) 

s  =  Sensitive  NGT  Question 

n  =  Nonsensitive  NGT  Question 

Comparison  of  Nx  with  Ny  Groups 

H-j  suggested  that  experimental  groups  would  attain  a  higher  level 
of  perceived  group  satisfaction  than  the  control  groups  on  each  of  the 
individual  satisfaction  variables.    The  data  suggest  that       is  true 
for  two  satisfaction  variables  (see  Table  1).    Experimental  groups  were 
more  satisfied  that  ideas  they  presented  could  not  be  "traced  back"  to 
them.    As  one  of  the  prime  reasons  for  Mod  NGT  was  to  avoid  the  linking 
of  ideas  and  individuals,  this  finding  is  a  confirmation  that  this 
linkage  is  more  difficult  under  Mod  NGT.    N    groups  were  also  more 
satisfied  with  the  "quantity  of  ideas  produced"  by  the  group.  Although 
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TABLE  1 

COMPARISON  OF  Nx  GROUPS  WITH  Ny  GROUPS  ON 
PERCEIVED  GROUP  SATISFACTION 


Sati  s . 
Variable 

Nx  (N=72) 
Rank  Sum 

Ny  (N=66) 
Rank  Sum 

Kruskal 
Test  Stat. 

Level 
Sign. 

1  Traced 

5681 

3909 

8.5 

.0034* 

2  Defend 

5221 

4375 

.9 

.34 

3  Pass 

5136 

4454 

.44 

.50 

4  Disagree 

5091 

4500 

.15 

.69 

5  Clarify 

4849 

4741 

.48 

.49 

6  Consider. 

4971 

4620 

.02 

.89 

7  Opport. 

5289 

4301 

1.6 

.20 

o  1 1  me  LIT. 

OUOD 

1  9 
.  1  C. 

9  Not  Dom 

5234 

4357 

1 .01 

.31 

10  Grp  Lead. 

4993 

4597 

.001 

.96 

11  All  Proc. 

5001 

4590 

0.0 

1.00 

12  Quant. 

5817 

3773 

12.7 

.0004* 

13  Qual. 

5088 

4502 

.13 

.71 

14  Fair 

5215 

4375 

.85 

.36 

15  Yours 

5037 

4553 

.02 

.89 

16  Goals 

5147 

4444 

.39 

.53 

*Significant  at  the  .05  level  (two  tailed) 
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none  of  the  other  variables  were  statistically  significant,  all 
fourteen  did  have  higher  rank  sums  than  the  control  groups, 

Hr,  suggested  that  the  N    groups  would  receive  a  higher  consensual 

L.  A 

judgement  rating  for  its  "top-seven"  items  than  the       groups.  Cadre 
ACR  (Average  Cadre  Ranking)  rankings  offered  no  significant  evidence 
to  support  that  conclusion.    Based  on  the  assumption  that  the  Cadre, 
with  its  years  of  experience,  would  be  best  able  to  judge  the  quality 
of  any  suggestion,  it  would  seem  that  the  N    groups  ideas  were  no 

A 

better  than  the       groups.    When  the  Cadre  (ACR)  rankings  were  weighted 
with  the  rating  of  the  group  (Top  item  =  7  points,  next  item  =  6  points, 
etc.)  or  with  the  number  of  votes  given  to  the  item  during  the  group 
vote,  the  results  were  still  not  significant. 

However,  when  student  cadet  officers  rankings  (ASR)  were  used, 

the  N    groups  were  significantly  better  than  the  N    groups,  (see 

X  y  ■  • 

Table  2).    Student  (ASR)  rankings  weighted  with  points  or  group  votes 
failed  to  produce  significant  results.    Although  not  significant 
statistically,  each  of  the  remaining  six  rank  sums  was    higher  for  the 
experimental  groups  than  for  the  control  groups. 

states  that  the  experimental  groups  will  produce  a  larger 
number  of  ideas  than  the  control  group.    Statistically  that  hypothesis 
must  be  rejected  as  neither  the  net  quantity  (total,  less  crossed 
out  items)  nor  the  total  quantity  (net,  plus  crossed  out  items)  proved 
to  be  significant  (see  Table  3).    Crossed  out  items  were  added  back  in, 
based  on  the  assumption  that  the  combining  of  items  is  not  a  group 
judgement  of  low  quality  but  rather  to  join  similar  items  to  simplify 
the  voting  process. 
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TABLE  2 

COMPARISON  OF  Nx  GROUPS  WITH  Ny  GROUPS  ON 


CONSENSUAL  JUDGEMENT  SCORES 

Nx  (N=84)     X     Ny  (N=84)  Kruskal  Level 

Variable  Rank  Sum  Rank  Sum  Test  Stat.  Sign. 

Aver.  Cad. 

Ranking  (ACR)         7304  6891  .43  .51 

ACRx  Points           7198  6998  .10  .75 

ACR X  Votes             7273  6922  .30  .58 
Aver.  Stu. 

Ranking  (ASR)         7776  6420  4.6  .0305* 

ASRx  Points            7418  6778  1.03  .31 

ASR X  Votes             7645  6550  3.01  .08 
Num.  Votes 

for  Item                7235  6960  .19  .66 


♦Significant  at  the  .05  level  (two  tailed) 
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Research  by  Cooley  (1977)  indicated  that  some  members  of  Nominal 
groups  withhold  ideas.    The  Data  (see  Table  4)  seem  to  indicate  that 
all  groups  withhold  to  some  degree  but  that  the       groups  withhold 
significantly  less  than  the       groups  confirming  H-jq.    Converting  the 
variable  to  a  percentage  of  the  net  items  generated  does  not  yield  any 
statistically  significant  results.    Similarly  an  analysis  of  Table  5, 
"other"  variables,yields  no  additional  statistically  significant  results. 

Comparison  of  Nye;  with  Nyc;  Groups 

The  data  indicate  that  N     groups  are  more  satisfied  than  control 
groups  using  a  sensitive  question  on  three  variables  (see  Table  6). 
They  are  "traced  back,"  "time  efficient,"  and  satisfaction  with  the 
"quantity  of  ideas  produced."    All  twelve  other  variables  moved  in  the 
expected  direction  but  none  were  significant. 

All  three  of  the  cadre  consensual  judgement  ranking  scores  (ACR, 
ACRx Points,  ACR x Votes)  were  not  significant  between  N     and  N  but 
the  student  ranking  (ASR)  and  the  student  rankings  (ASR)  weighted  by 
the  group  vote  were  both  significant  indicating  that  the  N  groups 

AO 

were  superior  to  the  N     groups  on  these  consensual  judgement  measures 
(see  Table  7). 

Comparing  the  N     and  N     groups  on  quantity  provided  no  signifi- 
Xo  y  b 

cant  data  (see  Table  8).    The  comparison  of  the  groups  on  withheld 
items  yielded  no  significant  results  (see  Table  9).    The  comparison 
of  "other"  variables  also  failed  to  yield  any  significant  results 
(see  Table  10). 
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Variable 


TABLE  3 

COMPARISON  OF  Nx  GROUPS  WITH  Ny  GROUPS  ON 
QUANTITY  OF  IDEAS  GENE-RATED 


Nx  (N=12)  X 
Group  Mean 


Ny  (N=12) 
Group  Mean 


ANOVA 
F  Value 


Level 
Sign. 


Net  Quantity  26.92 

Total  (Crossed 

Out  and  Net)  34.33 


22.83 


26.50 


1 .05 


3.90 


32 


,06 


Variable 


TABLE  4 

COMPARISON  OF  Nx  GROUPS  WITH  Ny  GROUPS  ON 
WITHHELD  ITEMS 


Nx  (N=12) 
Group  Mean 


Ny  (N=12) 
Group  Mean 


ANOVA 
F  Value 


Level 
Sign. 


Number  Withheld 

%  Withheld 
of  Net 


3.08 


9.30 


13.7 


16.25  39.58  3.22 

*Significant  at  the  .05  level  (tail  area  probability) 


.0014^ 


.086 


TABLE  5 


COMPARISON 

OF  Nx  GROUPS  WITH 
OTHER  VARIABLES 

Ny  GROUPS  ON 

Variable 

Nx  (N=12) 
Group  Mean 

X    Ny  (N=12) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Elapsed  Time 

99.17 

86.58 

2.99 

.098 

#  Crossed  Out 

7.4 

3.75 

2.39 

.14 

%  Crossed  Out 

21.5 

14.0 

1 .25 

.28 

§  of  Votes  Taken 

1.333 

1.5 

.64 

.42 
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TABLE  6 

COMPARISON  OF        GROUPS  WITH  Nys  GROUPS  ON 
PERCEIVED  GROUP  SATISFACTION 


Satis . 
Variable 

Nxs  (N=36) 
Rank  Sum 

Nys  (N-30) 
Rank  Sum 

Kruskal 
Test  Stat. 

Level 
Sign. 

1  Traced 

1436 

775 

9.2 

.0024* 

2  Defend 

1276 

935 

.90 

.34 

3  Pass 

1174 

1037 

.26 

.61 

4  Disagree 

1221 

990 

.04 

.83 

5  Clarify 

1113 

1098 

1 .57 

.21 

6  Cnn<;idpr 

1137 

1073 

81 

3n 
.  ou 

7  Opport. 

1224 

986 

.06 

.80 

8  Time  Eff. 

1386 

824 

5.5 

.0192* 

9  Not  Dom. 

1331 

879 

2.85 

.0913 

10  Grp.  Lead. 

1302 

909 

1 .76 

.18 

11  All  Proc. 

1311 

899 

1.92 

.16 

12  Quant. 

1362 

849 

4.26 

.0388* 

13  Qual. 

1263 

947 

.58 

.45 

14  Fair 

1225 

986 

.06 

.80 

15  Yours 

1224 

986 

.06 

.80 

16  Goals 

1219 

992 

.03 

.86 

*Significant  at  the  .05  level  (two  tailed) 
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TABLE  7 

COMPARISON  OF  Nxs  GROUPS  WITH  Nys  GROUPS  ON 


CONSENSUAL  JUDGEMENT  SCORES 

Nxs  (N=42)  X    Nys  (N-42)  Kruskal  Level 

Variable  Rank  Sum  Rank  Sum  Test  Stat.  Sign. 

Aver.  Cad. 

Ranking  (ACR)        1810  1759  .05  .81 

ACRx  Points            1768  1802  .02  .88 

ACR X  Votes             1889  1680  ■       .87  .35 

Aver.  Stu. 

Ranking  (ASR)        2095  1474  7.8  .0052* 

ASRx  Points           1977  1593  2.95  .086 

ASR X  Votes  2169  1400  11.8  .0006* 
Num.  Votes 

for  Item                1995  1575  3.56  .0595 


*Signif1cant  at  the  .05  level  (two  tailed) 
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TABLE  8 


COMPARISON  OF  Nxs  GROUPS  WITH  Nyg  GROUPS  ON 
QUANTITY  OF  IDEAS  GENERATED 


Vari  able 

Nvc  (N=6) 
Group  Mean 

X 

Nvs  {N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Net  Quantity 

24.17 

23.67 

.01 

.94 

Total  (Crossed 
Out  and  Net) 

33.00 

29.66 
TABLE  9 

.30 

.59 

COMPARISON 

OF 

Nxs  GROUPS  WITH 
WITHHELD  ITEMS 

Nys  GROUPS  ON 

Variable 

Nxs  {N=6) 
Group  Mean 

X 

Nys  (N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Si  gn. 

Number  Withheld 

3.33 

8.29 

3.21 

.098 

%  Withheld  of 
Net 

20.67 

23.83 
TABLE  10 

.034 

.86 

COMPARISON 

OF 

Nxs  GROUPS  WITH 
OTHER  VARIABLES 

Nys  GROUPS  ON 

Variable 

Nvc  (N=6) 
Group  Mean 

X 

Nuc  (N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Elapsed  Time 

88.0 

92.67 

.39 

.54 

#  Crossed  Out 

8.83 

6.0 

.52 

.48 

1  Crossed  Out 

26.8 

19.1 

.54 

.48 

#  of  Votes 
Taken 

1.5 

1.67 

.29 

.60 
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Comparison  of  Nxnwith  NynGroups 

The  comparison  of        and  N     on  satisfaction  indicated  that  N 
groups  were  significantly  more  satisfied  with  the  "quantity"  (see 
Table  11)  than  the  N^^  groups.    Consensual  judgement  ratings  were 
compared  for  N^^  and  N^^  groups;  no  significant  results  were  obtained 
(see  Table  12).    Comparison  on  the  quantity  of  items,  however,  indicates 
that  the  experimental  groups  perform  significantly  better  than  the 
control  groups  when  the  total  number  of  items  are  compared  (see  Table  13) 

The  data  on  withheld  items  indicate  (see  Table  14)  that  the  N 

A  J 

groups  significantly  outperform  the  N^^  groups  on  both  the  number  and 
the  percent  of  items  withheld  (with  a  fewer  number  of  withheld  items 
being  the  more  desirable).    Analysis  of  "other"  variables  indicates  that 
the  Mod  NGT  groups  sessions  lasted  significantly  longer  than  the 
control  groups  (see  Table  15). 

Comparison  of  Nyc;  with  Nyp  Groups 

It  was  generally  expected  that  experimental  groups  using  sensitive 
questions  would  perform  at  the  same  level  (there  would  be  no  significant 
differences)  as  experimental  groups  using  a  nonsensitive  question.  This 
would  occur  because  the  experimental  treatment  would  eliminate  any 
inhibitions  raised  by  a  sensitive  question  thus  making  the  sensitive  and 
the  nonsensitive  treatments  essentially  equal.    On  the  dependent 
variable  satisfaction,  this  did  not  prove  to  be  true  for  the  measures 
"traced  back"  and  the  "not  dominated"  because  in  both  cases  the  N 

A  J 

groups  significantly  outperformed  the  N     groups.    The  remaining  (see 

A 1 1 

Table  16)  variables  show  no  significant  variation  in  either  direction. 
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TABLE  n 

COMPARISON  OF  Nxn  GROUPS  WITH  Nyn  GROUPS  ON 
PERCEIVED  GROUP  SATISFACTION 


oa  L  1  o . 

Variable 

''xn  JO/ 
Rank  Sum 

Rank  Sum 

Krijs  ka  1 
Test  Stat. 

Level 
Sign. 

1     1  r  dLc (J 

1  H  1  J 

1 21 

1  32 

.  25 

?  nofonri 
^   Uc  1  ciiU 

1  TlS 
1  O  J  J 

1  27'^ 

.24 

.63 

0  r aoo 

141  ? 

1215 

1 .63 

.  20 

H  u  1  ->a  y  1  cc 

1 

1  275 

20 

.65 

3  Lr  1  d  1  1  1  y 

1  OH  0 

1  1 

>      1  1 

72 

6  Consider 

1371 

1256 

.44 

.  50 

7  Opport. 

1453 

1174 

2.7 

.1099 

8  Time  Eff. 

1168 

1459 

2.7 

.098 

9  Not  Dom. 

1279 

1348 

.16 

.69 

10  Grp.  Lead. 

1192 

1435 

2.11 

.15 

11  All  Proc. 

1216 

1412 

1 .28 

.26 

12  Quant. 

1566 

1062 

8.5 

.0035* 

13  Qual. 

1296 

1333 

.048 

.83 

14  Fair 

1395 

1233 

.87 

.35 

15  Yours 

1322 

1305 

.01 

.92 

16  Goals 

1362 

1266 

.30 

.58 

*Significant  at  the  .05  level  (two  tailed) 
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TABLE  12 

COMPARISON  OF  Nxn  GROUPS  WITH  Nyp  GROUPS  ON 


CONSENSUAL  JUDGEMENT  SCORES 

Nxn  (N=42)    x    Ny^  (N=42)  Kruskal  Level 

Variable  Rank  Sum  Rank  Sum  Test  Stat.  Sign. 

Aver.  Cad. 

Ranking  (ACR)        1860  1710  .46  .50 

ACRx  Points           1855  1715  .39  .53 

ACR X  Votes  1782  1878  .00  .98 
Aver.  Stu. 

Ranking  (ASR)        1830  1739  .17  .68 

ASRx  Points           1770  1800  .12  .89 

ASRx.Votes  1709  1861  .46  .50 
Num.  Votes 

for  Item               1675  1895  .97  .32 


*Significant  at  the  .05  level  (two  tailed) 


TABLE  13 


COMPARISON  OF  Nxn  GROUPS  WITH  Nyn  GROUPS  ON 
QUANTITY  OF  IDEAS  GENERATED 


Nxn  (N=6) 
Variable            Group  Mean 

X 

Nyn  (N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Net  Quantity  29.67 

22.00 

1 .93 

.19 

Total  (Crossed 

Out  and  Net)  35.67 

23.5 

5.85 

.0361* 

*Significant  at  the  .05  level 

(tail  area  probability). 

TABLE  14 

COMPARISON 

OF  Nxn  GROUPS  WITH 
WITHHELD  ITEMS 

Nyn  GROUPS 

ON 

Nxn  (N-6) 
Variable            Group  Mean 

X 

Nyn  (N-6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Number  Withheld  2.83 

10.00 

9.23 

.0125* 

%  Withheld  of 

Net  11.83 

55.33 

5.62 

.0393* 

*Significant  at  the  .05  level 

(tail  area  probability). 

TABLE  15 

COMPARISON 

OF  Nxn  GROUPS  WITH 
OTHER  VARIABLES 

Nyn  GROUPS 

ON 

Nxn  (N-6) 
Variable            Group  Mean 

X 

Nyn  (N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Elapsed  Time  110.33 

80.5 

7.85 

.0187* 

#  Crossed  Out  6.0 

1.5 

3.07 

.11 

%  Crossed  Out  16.33 

8.83 

.78 

.40 

#  of  Votes  Taken  1.17 

1.33 

.38 

.55 

*Significant  at  the  .05  level 

(tail  area  probability). 
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TABLE  16 

COMPARISON  OF  Nxs  GROUPS  WITH        GROUPS  ON 
PERCEIVED  GROUP  SATISFACTION 


Satis 
Variable 

Nxs  (N=36) 
Rank  Sum 

Nxn  (N=36) 
Rank  Sum 

Kruskal 
Test  Stat. 

Level 
Sign. 

1  Traced 

1 533 

1094 

6.4 

.0111* 

2  Defend 

1352 

1275 

.21 

.64 

3  Pass 

1301 

1327 

.03 

.86 

4  Disagree 

1329 

1298 

.03 

.85 

5  Clarify 

1206 

1421 

1.6 

.20 

6  Consider 

1225 

1403 

1.1 

.30 

7  Opport. 

1197 

1431 

1.9 

.17 

8  Time  Eff. 

1435 

1193 

1.9 

.17 

9  Not  Dom. 

1496 

1131 

4.54 

.0331* 

10  Grp  Lead. 

1436 

1191 

2.15 

.14 

11  All  Proc. 

1348 

1279 

.16 

.69 

12  Quant. 

1273 

1354 

.23 

.63 

13  Qual. 

1369 

1258 

.41 

.52 

14  Fair 

1338 

1289 

.08 

.78 

15  Yours 

1297 

1331 

.04 

.85 

16  Goals 

1346 

1282 

.14 

.71 

♦Significant  at  the  .05  level  (two  tailed) 
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For  the  consensual  judgement  rankings,  the  data  reveal  no 
significant  results  for  the  Cadre  but  the  student  judges  ranking  (ASR) 
weighted  with  the  number  of  votes  for  the  item,  indicate  that  the 
groups  using  the  sensitive  question  were  superior.    The  number  of  votes 
per  item  was  also  significant  in  favor  of  the  sensitive  question.  This 
indicates  that  N     groups  gave  each  of  their  "top-seven"  items  a  higher 

A  O 

vote  total  than  the  N     groups  (see  Table  17). 

Neither  of  the  dependent  variables  quantity  (see  Table  18)  nor 
withheld  items  (see  Table  19)  yielded  any  significant  results.  An 
examination  of  "other"  variables,  however,  indicated  that  N^^  groups 
used  significantly  more  time  than  the  N     groups  (see  Table  20). 

Comparison  of  Nys  with  Ny^  Groups 

The  author  hypothesized  that  N^^  groups  would  perform  better  than 
%s  9'^°^P^  because  the  Delbecq  NGT  still  contains  some  aspects  that 
tend  to  inhibit  participants.    Participants  using  a  nonsensitive  ques- 
tion would  be  less  concerned  because  the  question  itself  was  less 
threatening. 

An  examination  of  the  data  on  satisfaction  seem  to  uphold  that 
hypothesis.    The  groups  using  the  nonsensitive  question  performed 
significantly  better  on  the  variables  "time  efficient"  and  "satisfied 
with  all  processes"  of  the  remaining  variables  moved  in  the  expected 
direction  (see  Table  21).    No  significant  results  were  obtained  from 
the  consensual  judgement  data  (see  Table  22). 

None  of  the  variables  under  "quantity"  (see  Table  23)  or  "withheld 
items"  (see  Table  24)  were  significant.    Comparing  "other"  variables, 
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TABLE  17 

COMPARISON  OF  Nxs  GROUPS  WITH  Nxn  GROUPS  ON 
CONSENSUAL  JUDGEMENT  SCORES 

Nxs  (N=42)    X    Nxn  (^=42)  Kruskal  Level 


Va  n"  hi  p 

V  U  1    1  U  U  1  c 

r\a  1 1  IS,  o  uiii 

Tpcf  9t;3t 
1  CO  U    O  La  L  . 

0 1  yn . 

Aver.  Cad. 
Ranking  (ACR) 

1741 

1829 

.16 

.69 

ACR  X  Points 

1742 

1828 

.15 

.70 

ACR  X  Votes 

1981 

1589 

3.07 

.08 

Aver.  Stu. 
Ranking  (ASR) 

1827 

1743 

.14 

.71 

ASR  X  Points 

1851 

1718 

.35 

.55 

ASR  X  Votes 

2076 

1494 

6.78 

.0092* 

Num.  Votes 
for  Item 

2158 

1411 

11.2 

.0008* 

*Significdnt  at  the  .05  level  (two  tailed) 
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TABLE  18 

COMPARISON  OF  Nxs  GROUPS  WITH  Nxn  GROUPS  ON 
QUANTITY  OF  IDEAS  GENERATED 

Nxs  (N=6)     X    Nxn  (N=6)            ANOVA  Level 
Variable  Group  Mean        Group  Mean  F  Value  Sign. 

Net  Quantity  24.17  29.67  .57  .47 

Total  (Crossed 

Out  and  Net)  33.0  35.67  .14  .71 


TABLE  19 


COMPARISON 

OF  Nxs  GROUPS  WITH 
WITHHELD  ITEMS 

Nxn  GROUPS  ON 

Variable 

Nxs  (N=6) 
Group  Mean 

X  Nxn 

Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Number  Withheld 

3.33 

2.83 

.05 

.82 

%  Withheld  of 
Net 

20.66 

11 .83 
TABLE  20 

.32 

.58 

COMPARISON  OF  Nxs  GROUPS  WITH  Nxn  GROUPS  ON 
OTHER  VARIABLES 

Variable 

Nxs  (N=6) 
Group  Mean 

X  Nxn  (N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Elapsed  Time 

88.00 

110.33 

5.32 

.0437* 

#  Crossed  Out 

8.83 

6.0 

.41 

.53 

%  Crossed  Out 

26.8 

16.33 

.84 

.38 

#  of  Votes  Taken 

1.5 

1.17 

1 .42 

.26 

*Significant  at  the  .05  level  (tail  area  probability) 
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TABLE  21 


COMPARISON  OF  Nys  GROUPS  WITH  Nyp  GROUPS  ON 
PERCEIVED  GROUP  SATISFACTION 


Satis. 
Vari  able 

Nys  (N=30) 
Rank  Sum 

Nys  (N=36) 
Kank  Sum 

Kruskal 
Test  Stat 

Level 

Sinn 

1  Traced 

1052 

1158 

.38 

.54 

2  Defend 

1010 

1201 

.00 

.95 

3  Pass 

1108 

1103 

2.38 

.12 

4  Disagree 

1023 

1187 

.06 

.80 

5  Clarify 

1023 

1187 

.06 

.80 

6  Consider 

1050 

1160 

.36 

.55 

7  Opport. 

995 

1215 

.02 

.89 

8  Time  Eff. 

784 

1427 

8.25 

.0041* 

9  Not  Dom. 

982 

1229 

.09 

.76 

10  Grp.  Lead. 

908 

1302 

1 .74 

.18 

n  All  Proc. 

846 

1364 

4.36 

.0367* 

12  Quant 

1030 

1180 

.11 

.74 

13  Qual . 

978 

1233 

.13 

.72 

14  Fair 

1075 

1135 

.87 

.35 

15  Yours 

981 

1230 

.10 

.75 

16  Goals 

1060 

1151 

.52 

.47 

*Significant  at  the  .05  level  (two  tailed) 
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TABLE  22 

COMPARISON  OF  Nys  GROUPS  WITH  Nyn  GROUPS  ON 


CONSENSUAL  JUDGEMENT  SCORES 

Nys  (N=42)  X    Nyn  (N=42)  Kruskal  Level 

Variable  Rank  Sum  Rank  Sum  Test  Stat.  Sign. 

Aver.  Cad. 

Ranking  (ACR)         1799  1771  .02  .90 

ACRx  Points           1832  1738  .17  .67 

ACR X  Votes  1883  1686  .78  .38 
Aver.  Stu. 

Ranking  (ASR)        1584  1986  3.3  .07 

ASRx  Points           1661  1908  1.2  .27 

ASR X  Votes  1673  1897  1.0  .32 
Num.  Votes 

for  Item                1834  1735  .20  .66 


*Significant  at  the  .05  level  (two  tailed) 
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TABLE  23 

COMPARISON  OF  Nys  GROUPS  WITH  Nyn  GROUPS  ON 


QUANTITY  OF  IDEAS  GENERATED 

Nys  (N=6)     X    Nyn  (N=5)             ANOVA  Level 
Variable  Group  Mean  Group  Mean  F  Value  Sign. 

Net  Quantity  23.67  22.00  .20  .66 

Total  (Crossed 

Out  and  Net)  29.67  23.5  3.01  .11 


TABLE  24 

COMPARISON  OF  Nys  GROUPS  WITH  Nyn  GROUPS  ON 


WITHHELD  ITEMS 

Nys  (N=6)      X    Nyn  (N=6)             ANOVA  Level 
Variable  Group  Mean  Group  Mean  F  Value  Sign. 

Number  Withheld        8.25  10.0  .39  .54 

%  Withheld  of 

Net  35.75  55.33  .72  .41 


TABLE  25 


COMPARISON 

OF  Nys  GROUPS  WITH 
OTHER  VARIABLES 

Nyn  GROUPS  ON 

Variable 

Nys  (N=6) 
Group  Mean 

X    Nyn  (N=6) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Elapsed  Time 

92.66 

80.5 

1 .99 

.19 

#  Crossed  Out 

6.0 

1.5 

7.54 

.02* 

%  Crossed  Out 

19.17 

8.83 

2.14 

.17 

#  of  Votes  Taken 

1.67 

1 .33 

1 .25 

.29 

*Significant  at  the  .05.  level  (tail  area  probability). 
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the  number  of  crossed  out  items  was  significantly  greater  for  groups 
using  the  sensitive  question  (see  Table  25)  than  the  nonsensitive 
question. 

Comparison  of  Ns  with  Nn  Groups 

It  was  generally  expected  that  the  nonsensitive  groups  would  do 
as  well  or  better  than  the  sensitive  groups  because  fewer  inhibitions 
would  be  raised  under  the  former  than  the  latter.    Data  on  perceived 
satisfaction  do  not  lend  much  support  to  that  hypothesis  as  the  group 
using  the  sensitive  question  significantly  outperformed  the  groups 
on  the  "traced  back"  variable  (see  Table  26).    The  consensual  judgement 
data  showed  no  significant  results  but  the  "number  of  votes"  for  the 
item  was  significantly  higher  for  groups  using  the  sensitive  question 
as  opposed  to  the  nonsensitive  question  (see  Table  27). 

No  significant  differences  were  noted  in  the  "quantity"  data  (see 
Table  28)  or  in  the  number  of  "withheld  items"  (see  Table  29).  Like- 
wise no  significant  results  were  obtained  when  comparing  the  "other" 
variables  for      and       groups  (see  Table  30). 

Additional  Data 

In  addition  to  the  data  gathered  on  the  dependent  variables,  some 
other  data  were  collected  including: 

Elapsed  Time--of  the  six  between-group  comparisons,  only 
two  resulted  in  significant  differences  in  the  amount  of 
time  used  for  the  NGT  session.    (1)  The  N^^  groups  used 
significantly  more  time  than  the  N^^  groups  and  (2)  the 
N     groups  used  significantly  more  time  than  the  N  groups. 
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TABLE  26 

COMPARISON  OF  N5  GROUPS  WITH       GROUPS  ON 
PERCEIVED  GROUP  SATISFACTION 


bat! s . 
Variable 

Ns  (N=d5) 
Rank  Sum 

\  (N-72) 
Rank  Sum 

Kruskal 
Test  Stat. 

Level 
Siqn. 

1  Traced 

c  0  n  "7 

5097 

4493 

4.87 

.0272* 

c  uetena 

/I  r  TO 

4672 

4919 

.14 

.70 

6  raSS 

4782 

4809 

.97 

.32 

4  Disagree 

4634 

4956 

.04 

.83 

0  L 1  an  fy 

4402 

5188 

.68 

.40 

6  Consider 

4524 

5066 

.07 

•79 

7  Opport. 

4347 

5244 

1.12 

.29 

8  Time  Eff. 

4403 

5188 

.63 

.43 

9  Not  Dom. 

4919 

4671 

2.11 

.15 

10  Grp.  Lead. 

4621 

4970 

.02 

.87 

n  All  Proc. 

4332 

5258 

1.23 

.27 

12  Quant. 

4603 

4987 

.01 

.94 

13  Qual. 

4644 

■  4946 

.06 

.80 

14  Fair 

4791 

4799 

.79 

.37 

15  Yours 

4506 

5084 

.12 

.72 

16  Goals 

4785 

4806 

.74 

.39 

*Significant  at  the  .05  level  (two  tailed) 
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TABLE  27 

COMPARISON  OF  Ng  GROUPS  WITH  Np  GROUPS  ON 
CONSENSUAL  JUDGEMENT  SCORES 


Ns  (N=84)    X     Nn  (N=84)  Kruskal  Level 


a  hi  o 

V  U  1    1  U  U  1  c. 

KanK  oum 

lest  Otclt. 

Sign. 

Aver.  Cad. 
Ranking  (ACR) 

7034 

7162 

.04 

.84 

ACR  X  Points 

7111 

7085 

.00 

.97 

ACR  X  Votes 

7657 

6539 

3.1 

.08 

Aver.  Stu. 
Ranking  (ASR) 

6816 

7380 

.81 

.37 

ASR  X  Points 

6987 

7209 

.12 

.72 

ASR  X  Votes 

7437 

6759 

1.16 

.28 

Num.  Votes 
for  Item 

7905 

6290 

6.6 

.0103* 

*Significant  at  the  .05  level  (two  tailed) 
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TABLE  28 

COMPARISON  OF       GROUPS  WITH  Np  GROUPS  ON 
QUANTITY  OF  IDEAS  GENERATED 

Ns  (N=12)      X    Nn  {N=12)             ANOVA  Level 
Variable  Group  Mean  Group  Mean  F  Value  Sign. 

Net  Quantity  23.92  25.83  .22  .64 

Total  (Crossed 

Out  and  Net)  31 .33  29.58  .17  .68 


TABLE  29 

COMPARISON  OF  Ng  GROUPS  WITH  Nn  GROUPS  ON 
WITHHELD  ITEMS 

Ns  (N=10)      X    Nn  (N=12)             ANOVA  Level 
Variable  Group  Mean        Group  Mean  F  Value  Sign. 

Number  Withheld        5,30  6.4  .26  .61 

%  Withheld  of 

Net  22.2  33.6  .68  .41 


TABLE  30 


COMPARISON 

OF  Ns  GROUPS  WITH  Np 
OTHER  VARIABLES 

GROUPS  ON 

Variable 

Ns  (N=12) 
Group  Mean 

X    Np  (N=12) 
Group  Mean 

ANOVA 
F  Value 

Level 
Sign. 

Elapsed  Time 

90.33 

95.4 

.44 

.52 

#  Crossed  Out 

7.4 

3.75 

2.38 

.14 

%  Crossed  Out 

23.0 

12.5 

2.48 

.129 

#  of  Votes  Taken 

1 .58 

1.25 

2.83 

.11 

Number  of  Crossed  Out  Itenis--the  number  of  items  recorded 
on  the  flip  charts  and  later  crossed  out  by  the  group  were 
tabulated.    The  data  indicate  that  the  only  significant 
difference  was  between  N^^  and  the  N^^  groups,  with  the  N^^ 
groups  crossing  out  significantly  more  items  than  the 
^yn  groups. 

Participants  Subjective  Comments--each  participant,  as  part 
of  the  satisfaction  questionnaire,  was  asked  to  describe  the 
aspects  of  the  NGT  process  he/she  liked  most  and  least. 
For  the  experimental  groups,  the  most  liked  aspects  were  the 
discussion  phase  (32),  the  round  robin  (21),  the  voting  phase 
(10),  and  other  categories  (9).    For  the  Delbecq  NGT  groups 
the  most  liked  aspects  were  the  discussion  phase  (34),  those 
who  did  not  answer  (15),  the  round  robin  (12)  and  other 
categories  (12). 

The  least  liked  aspects  for  the  Mod  NGT  groups  were 
time  (25),  discussion  phase  (12),  round  robin  (5),  and  other 
(26).    The  least  liked  aspects  for  the  Delbecq  NGT  groups 
were  discussion  phase  (18),  did  not  answer  (17),  time  (12), 
round  robin  (6),  and  other  categories  (19). 
Recorder  Comments--Group  recorders  were  asked  to  comment  on 
any  problems  they  encountered  during  the  NGT  session.  Of 
the  twenty-four  groups  only  five  recorders  made  written 
comments.    Two  recorders  discussed  problems  with  the  tape 
recorders  while  two  others  noted  that  it  was  difficult  to 
keep  the  members  interested  as  the  session  progressed.  One 
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recorder  explained  why  he  felt  his  group  had  crossed 
out  so  many  ideas  (because  the  ideas  improved  as  the 
members  became  more  involved). 


1 


VII.      CONCLUSIONS  AND  SUGGESTIONS  FOR  FUTURE  RESEARCH  i 

i 

i 

Conci usions 

As  mentioned  previously,  this  study  was  designed  to  test  the 
hypotheses  regarding  whether  (1)  Mod  NGT  groups  would  outperform  the 
Delbecq  NGT  groups  and  (2)  whether  the  relative  sensitivity  of  the 
NGT  question  has  any  effect  on  group  performance.    Given  the  small  N 
and  the  relative  similarity  between  the  NGT  Mod  and  the  Delbecq 
techniques,  as  well  as  the  limited  degree  in  which  the  two  NGT  ques- 
tions differ  on  relative  sensitivity,  the  author  did  not  expect  large 
statistical  differences  for  all  variables  between  each  of  the  treat- 
ments.   Thus  even  though  the  majority  of  this  chapter  will  be  devoted 
to  variables  that  are  statistically  significant,  some  comments  on 
indicators  or  tendencies  (for  nonsignificant  variables)  will  also 
be  included  in  the  discussion. 

Considering  each  of  the  four  major  dependent  variables  (satisfac- 
tion, consensual  judgement  score,  quantity  and  withheld  items)  the 
following  conclusions  were  reached. 

Comparison  of  Ny.  with  Ny  Groups 

It  was  predicted  (H^,  H^,  H^,  H^q)  that  the  Mod  NGT  groups  would 
perform  better  than  the  Delbecq  NGT  groups.    The  data  indicate  that 
the  Mod  NGT  groups  did  tend  to  perform  better.    Four  of  the  measures 
were  statistically  significant  revealing  the  superiority  of  the  Mod  NGT 
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on  (1)  the  number  of  "withheld  items,"  (2)  the  Average  Student 
Ranking  (ASR)  consensual  ranking  scores,  (3)  satisfaction  that  items 
presented  could  not  be  "traced  back,"  and  (4)  satisfaction  with  the 
"quantity  of  items  produced."    This  study  provides  no  evidence  to 
suggest  that  the  Delbecq  NGT  is  superior  to  the  Mod  NGT.    Not  one  of 
the  remaining  tv/enty-two  dependent  measures  supported  the  superiority 
of  the  Delbecq  technique  (control  group).    In  fact  all  of  the  remaining 
dependent  measures  (although  nonsignificant)  moved  in  the  direction  of 
the  superiority  of  the  Mod  NGT. 

Mod  NGT  groups  withheld  fewer  ideas  thus  giving  support  for  the 
contention  that  the  Mod  NGT  process  eliminates  some  of  the  inhibitions 
present  in  the  Delbecq  NGT.    Mod  NGT  groups  were  more  satisfied  with 
the  "quantity  of  items  produced"  by  the  group  and  with  the  fact  that 
an  item  presented  to  the  group  could  not  be  "traced  back"  to  its 
originator.    The  latter  indicates  that  the  Mod  NGT  was  better  able  to 
separate  the  idea  from  the  individual  presenting  it.    The  elimination 
of  this  linkage  is  a  desirable  characteristic  if  ideas  are  to  be 
evaluated  on  merit,  rather  than  on  the  status  of  the  individuals 
presenting  them.    Mod  NGT  groups  also  generated  items  that  received 
higher  consensual  judgement  rankings  (ASR)  from  the  student  judges 
than  items  generated  by  the  Delbecq  NGT  groups  received. 

Comparison  of  Nxs  with  Nyt;  Groups 

The  author  predicted  (H^,  H^,  Hg)  that  the  experimental  groups 
using  a  sensitive  question  would  outperform  the  control  groups  using 
a  sensitive  question.    The  data  indicate  that  the  N     groups  do  tend 
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to  outperform  the  N     groups.    Five  of  the  dependent  measures  were 
statistically  significant  indicating  that  the  N^^  outperformed  the 
N     on  (1)  satisfaction  that  ideas  couldn't  be  "traced  back,"  (2) 
satisfaction  that  the  process  was  "time  efficient,"  (3)  satisfaction 
with  the  "quantity"  of  ideas  generated  by  the  group,  (4)  the  ASR 
student  consensual  judgement  ranking,  and  (5)  the  ASR  ranking  weighted 
by  the  votes  cast  for  the  item  by  the  group.    None  of  the  dependent 
measures  were  significant  in  the  opposite  direction,  meaning  that  in 
no  case  did  the  N^^  groups  give  significantly  better  results  than  the 
^xs  9'^°^PS-    I"  fact,  twenty  of  the  remaining  twenty-one  dependent 
measures  (all  nonsignificant)  tended  toward  the  direction  of  supporting 
the        groups  as  being  superior. 

Comparison  of  Nyn  with  Nyn  Groups 

It  was  predicted  (H-^,  Hg,  Hg)  that  the  N^^groups  would  outperform 
the  N^^  groups  even  on  this  less  sensitive  question.    The  evidence 
indicates  that  the  N^^  groups  did     significantly    outperform  the 
N     groups  on  four  variables:    (1)  the  total  quantity  of  ideas  produced; 

y-* 

(2)  satisfaction  with  the  quantity  of  ideas  generated;  (3)  the  number 
of  items  withheld  from  the  group;  and  (4)  the  percent  of  items  that 
were  withheld. 

This  indicates  that  the  N     groups  generated  more  items,  were 
more  satisfied  with  the  number  they  produced  and  withheld  fewer  items 
than  did  the  N^^  groups.    In  no  case  did  the  N^^  groups  give  significantly 
better  results  than  the  N^^  groups. 
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Comparison  of  Nx<;  with  Nyn  Groups 

It  was  suggested  that  there  would  be  no  significant  differences 

(in  performance)  between  the  two  groups.    This  would  be  due  to  the 

fact  that  the  Mod  NGT  technique  should  eliminate  any  inhibiting  factors 

caused  by  a  sensitive  question,  thus  making  the  groups  essentially 

equal.    The  data,  however,  indicate  that  the  groups  are  not  equal 

and  that  the  N     groups  significantly  outperform  the  N  „  groups  on 
x->  xn 

three  variables— (1 )  satisfaction  that  an  idea  would  not  be  "traced 
back,"  (2)  satisfaction  that  the  group  was  "not  dominated"  by  anyone, 
and  (3)  the  student  (ASR)  consensual  judgement  ranking  weighted  by 
the  votes  given  each  item  by  the  group.    The  tendency  for  N  groups 
to  outperform  N^^  groups  could  be  a  result  of  the  fact  that  sensitive 
questions  can  be  stimulating  as  well  as  inhibiting.    Thus  with  the  Mod 
NGT  neutralizing  the  inhibiting  factors  the  end  result  could  be  a 
slight  tendency  towards  being  more  productive.    In  no  case  did  the  N 

xn 

groups  significantly  outperform  the  N  groups. 

A  o 

Comparison  of  Ny^  with  N^^  Groups 

Because  sensitive  questions  can  be  inhibiting,  it  was  suggested 
that  the  Delbecq  NGT,  with  its  inhibiting  aspects,  would  be  less  effective 
on  sensitive  questions  than  on  nonsensitive  ones.    The  analysis  of 
the  data  produced  evidence  that  the  N^^  groups  did  significantly  out- 
perform the  N     groups  on  two  dependent  measures:    (1)  satisfaction 
y-* 

with  all  aspects  of  the  process;  and  (2)  satisfaction  that  the  process 
was  time  efficient.    Both  of  these  variables  indicated  that  the  groups 
with  the  nonsensitive  question  outperformed  the  groups  with  a  sensitive 
question  as  predicted.    In  no  case  did  the  N     groups  significantly 
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outperform  the  N     groups.    This  tendency  for  N     groups  to  outperform 
groups  could  be  an  indicator  that  inhibitors  (raised  by  a  sensitive 
question)  are  not  being  adequately  reduced  by  the  Delbecq  technique. 
As  a  result,  groups  without  inhibitions  (those  using  nonsensitive  ques- 
tions) outperform  those  with  inhibiting  factors  (those  using  sensitive 
questions). 

Comparison  of      with  Nn  Groups 

It  was  suggested  that  groups  using  a  nonsensitive  question  would 
outperform  the  groups  using  sensitive  questions.    The  data,  however, 
indicate  that  the  N    groups  significantly  outperformed  the  N  groups 
on  the  satisfaction  variable    "traced  back."    In  no  case  did  the  N 

n 

groups  significantly  outperform  the  groups. 

Overall  Conclusions 

Of  the  seventy-eight  dependent  measures  used  to  compare  the  Mod 
NGT  groups  with  the  Delbecq  NGT  groups,  thirteen  (1  out  of  6)  were 
significant  at  the  .05  level.    All  thirteen  significant  measures 
supported  the  hypothesis  that  the  Mod  NGT  outperformed  the  Delbecq  NGT. 
In  no  case  did  the  Delbecq  NGT  significantly  outperform  the  Mod  NGT. 

The  Mod  NGT  groups  outperformed  the  Delbecq  NGT  groups  regardless 
of  whether  a  sensitive  or  a  nonsensitive  question  was  used.    The  Mod 

NGT  performed  better  (against  itself,  N     vs.  N    )  using  a  sensitive 

X  s  X  n 

NGT  question,  rather  than  a  nonsensitive  question.    On  the  other  hand, 
the  Delbecq  NGT  performed  better  (against  itself,  N     vs.  N     )  using  a 
a  nonsensitive  NGT  question,  rather  than  a  sensitive  one. 
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Suggestions  for  Future  Research 

To  provide  meaningful  results  from  experiments  that  simulate 
real  world  processes,  consideration  should  be  given  toward  using 
existing  as  opposed  to  ad  hoc  decision  groups.    Studying  the  longi- 
tudinal benefits  of  the  Mod  NGT  might  also  add  vital  information  to 
the  literature  on  small  group  decision  making.    Consideration  should 
also  be  given  to  comparing  the  Mod  NGT  with  conventional  interacting 
groups.    Although  there  are  numerous  studies  cited  in  Chapter  II 
demonstrating  the  relative  superiority  of  the  NGT,  their  failure  to 
adhere  to  the  NGT  procedures  makes  one  question  if  close  adherence  to 
procedures  might  not  provide  even  more  dramatic  results. 

Another  major  consideration  for  any  future  research  is  the 
relative  value  of  the  quality  or  consensual  judgement  rating.  Cooley 
(1977)  devotes  an  entire  chapter  to  the  problem  of  an  accurate 
quality  rating  without  resolving  the  issue.    The  consensual  judgement 
ranking  by  the  Cadre  failed  to  be  significant  in  any  of  the  eighteen 
possible  comparisons  while  the  student  ratings  proved  significant  four 
times.    This  could  be  an  indication  of  a  poor  agreement  on  relevant 
issues  between  the  Cadre  and  the  total  body  of  ROTC  students  that 
participated  in  this  study.    So,  even  though  they  have  more  experience 
and  knowledge  of  realistic  solutions,  it  appears  that  their  value  as 
raters  for  this  particular  ROTC  unit  must  be  questioned.    The  only 
plausible  explanation  is  that  the  lack  of  variance  (no  significant  high 
or  low  groups)  is  due  to  the  fact  that  their  high  and  low  rated  items 
were  the  average  items  for  the  cadets  in  the  NGT  sessions,  and  vice 
versa.    Although  this  quality  rating  problem  is  not  new,  it  follows 
that,  for  this  ROTC  unit,  peer  ratings  may  be  the  only  acceptable 


alternative  (rather  than  dropping  the  quality  measure  altogether) 
when  such  a  larger  number  of  items  (168)  must  be  rated.    In  addition 
an  alternative  to  alternation  ranking  (2-4  hours)  or  paired  compari- 
sons must  be  developed  to  ensure  accurate  ratings  of  large  quantities 
of  ideas. 


APPENDIX  A 


NOMINAL  GROUP  TECHNIQUE 


Delbecq  et  al .  (1975) 

Summary  Leadership  Guide  for  Conducting  an  NGT  Meeting. 

Preparing  the  Meeting  Room 
Table  Arrangement: 

Tables  arranged  as  an  open  "U"  with  a  flip  chart  at  the  open 
end  of  the  table 

Sufficient  space  between  tables  to  avoid  interference 
Supplies: 

Flip  chart  for  each  table  and  for  the  leader 
Roll  of  masking  tape 

Nominal  v/orksheet  and  pencils  for  each  participant 
3x5  cards  (for  ranking);  voting  forms  (for  rating) 
Felt  pens 

Introducing  the  Meeting 
Welcome  Statement: 

Cordial  and  warm  welcome 

Statement  of  the  importance  of  the  NGT  task 

Clarification  of  the  importance  of  each  group  member's  contribution 
Statement  of  the  use  or  purpose  of  the  meeting's  output 


Conducting  the  Nominal  Group  Process 
STEP  1.    SILENT  GENERATION  OF  IDEAS  IN  WRITING 
Process : 

Present  the  nominal  question  to  the  group  in  writing 
Verbally  read  the  question 

Illustrate  level  of  abstraction  and  scope  desired  with  example 
which  does  not  distort  (lead)  group  responses 
Avoid  other  requests  for  clarification 

Charge  the  group  to  write  ideas  in  brief  phrases  or  statements 
Ask  group  members  to  work  silently  and  independently 
Model  good  group  behavior 

Sanction  disruption  of  the  silent,  independent  activity  by 
comments  addressed  to  group  as  a  whole. 
Benefi  ts : 

Provides  adequate  time  for  thinking 

Facilitates  hard  work  by  the  model  of  other  group  members 

reflecting  and  writing 

Avoids  interrupting  each  other's  thinking 

Avoids  premature  focusing  on  single  ideas 
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Delbecq  et  al .  (1975) 
NGT  (cont.) 

Eliminates  dominance  by  high-status  or  agressive  members 

in  idea  generation 

Keeps  the  group  problem-centered 

STEP  2.    ROUND-ROBIN  RECORDING  OF  IDEAS  ON  A  FLIP  PAD 
Process : 

Provide  clear  instructions  concerning  the  step: 

Indicate  objective  of  the  step  is  to  map  the  group's  thinking 
Explain  need  to  present  ideas  in  brief  words  or  phrases 
Explain  process  of  taking  one  idea  serially  from  each  member 
Explain  group  members  must  decide  if  items  are  duplicates 
Explain  that  an  individual  may  "pass"  when  he  has  no  further 
items,  but  may  "reenter"  later 

Express  the  desirability  of  hitchhiking  and  adding  new 
ideas  even  if  they  are  not  on  individual  nominal  worksheets 
Explain  i nappropriateness  of  discussion  prior  to  completion 
of  listing 
Quick  effective  mechanical  recording: 
Record  ideas  as  rapidly  as  possible 
Record  ideas  in  the  words  used  by  group  members 
Provide  assistance  in  abbreviating  only  in  special  situations 
Make  the  entire  list  visible  by  tearing  off  completed  sheets 
and  taping  them  on  an  area  visible  to  all  group  members 
Sanction  group  as  a  whole  if  individuals  engage  in  side  conversations 
or  attempt  to  discuss  items  prior  to  completing  the  listing 
Benefi  ts : 

Equalizes  opportunity  to  present  ideas 
Assists  in  separating  ideas  from  personalities 
Provides  a  written  record  and  guide: 

Increases  group's  ability  to  deal  with  a  larger  number 

of  ideas 

Avoids  loss  of  ideas 

Confronts  the  group  with  an  array  of  clues 

Encourages  hitchhiking 
Places  conflicting  ideas  comfortably  in  front  of  group 
Forces  the  group  to  fully  explore  the  problem. 

STEP  3.    SERIAL  DISCUSSION  FOR  CLARIFICATION 
Process : 

Verbally  define  the  meaning  of  items 
To  clarify  the  meaning  of  items 
To  explain  reasons  for  agreement  or  disagreement 

Indicate  the  final  judgments  will  be  expressed  by  voting 

so  arguments  are  unnecessary 

Pace  the  group  so  that  all  ideas  receive  sufficient  time 
for  clarification 

Avoid  forcing  the  member  who  originally  lists  the  idea  to 
be  solely  responsible  for  clarifying  the  item 
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Delbecq  et  al .  (1975) 
NGT  (cont.) 

Benefits: 

Avoids  having  discussion  focus  unduly  on  any  particular 

idea  or  subset  of  ideas 

Helps  eliminate  misunderstanding 

Provides  opportunity  to  express  the  logic  behind  items 
Allows  members  to  disagree  without  undue  argumentation 

STEP  4.    PRELIMINARY  VOTE  ON  ITEM  IMPORTANCE 
Process: 

Ask  the  group  to  select  from  the  entire  list  a  specific 

number  (7  ±  2)  of  priority  (important)  items: 

Place  each  priority  item  on  a  3  x  5  card  or  rating  form 
Rank-order  or  rate  the  selected  priority  items 

Collect  the  cards  or  rating  forms  and  shuffle  them  to 

retain  anonymity 

Tally  the  vote  and  record  the  results  on  the  flip  chart  in 
front  of  the  group 
Benefits: 

Obtaining  independent  judgments  in  writing  helps  eliminate 
social  pressures 

Expressing  judgments  mathematically  by  ranking-ordering  or 
rating  increases  accuracy  of  judgments 

Displaying  the  array  of  individual  votes  clearly  highlights 
areas  needing  further  clarification  or  discussion 

STEP  5.    DISCUSSION  OF  THE  PRELIMINARY  VOTE 
Process: 

Define  the  role  of  the  step  as  clarification,  not  pressure 
toward  artificial  consensus 
Keep  the  discussion  brief 

Caution  group  members  to  think  carefully  about  any  changes 
they  make  in  their  voting 
Benefits : 

Provide  group  members  a  final  opportunity  to  clarify  their 
positions 

Ensures  that  "spread"  votes  really  reflect  differences  in 
judgment,  not  unequal  information  or  misunderstanding 

STEP  6.     FINAL  VOTE 
Process: 

(Repeat  Step  4. ) 
Benefi  ts : 

Accurate  aggregation  of  group  judgments  and  error  reduction 
Closure  to  the  meeting 

Taken  from:    Group  Techniques  for  Program  Planning,  Andre  L.  Delbecq, 
Andrew  H.  Van  de  Ven  and  David  H.  Gustafson.  Glenview, 
Illinois:    Scott,  Foresman  and  Co.,  1975. 


Group  Number 


APPENDIX  B 
MOD  NGT— INSTRUCTIONS  TO  GROUP  LEADERS 


Please  read  these  instructions  carefully.  If  you  have  any  questions, 
ask  one  of  the  assistants  to  help  you. 

Your  Function 

Your  job  as  group  leader  is  to  ensure  that  the  procedures  (as 
outlined  below)  for  the  Nominal  Group  Process  are  strictly 
adhered  to. 


Some  Do's  and  Don'ts 


Don't 

--contribute  your  own  ideas  or  solutions 
--criticize  ideas 
— allow  arguments 

--ask  the  initiator  of  an  idea  to  explain  it 
--suggest  compromises 
--reword  ideas  for  members 

Do 

--try  to  keep  the  process  from  bogging  down 
--write  swiftly  but  legibly  on  the  flip  chart  pages 
--tally  the  votes  swiftly 

--avoid  long  explanations  of  the  Nominal  Group  Process 
--try  to  stop  any  side  conversations  between  members 
--ask  for  assistance  if  you  have  any  questions 


Procedures  for  a  Group  Leader 


Set  Up 

A.  You  will  be  randomly  assigned  6  participants.    If  you  do  not 
have  six,  ask  for  assistance. 

B.  You  will  be  assigned  a  table  either  in  the  mess  hall  or  in  the 
rec.  hall  (to  the  east  of  the  mess  hall  towards  the  river). 
Participants  should  sit  in  a  "U"  shape  (three  to  a  side)  with  you 
at  the  head  of  the  table  with  the  large  recording  sheets. 

C.  Plug  in  the  tape  recorder  (if  assigned)  make  sure  it  works  and 
that  it  is  as  inconspicuous  as  possible. 
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D.    Hand  out  from  your  packet  to  each  participant 
1  pencil 

1  sheet  with  the  topic  question  typed  on  it 

2  individual  voting  sheets 
10  3  X  5  cards 


INTRODUCTION  AND  WELCOME 

A.  Purpose--In  friendly  terms  explain  that:    "You  are  participating 
in  a  dual  purpose  exercise.    The  first  is  to  provide  the  AROTC 
with  honest  feedback  from  its  cadets  on  the  topic  area  listed  on 
the  sheet  you  were  just  handed.    The  army  is  looking  for  creative 
but  practical  solutions  to  this  problem.    The  second  purpose  of 
this  exercise  is  to  acquaint  you  with  a  useful  leadership  tool 
for  group  decision  making--the  Nominal  Group  Technique.    As  a 
future  leader  you  may  find  that  this  technique  might  have  applica- 
tions in  many  facets  of  army  life." 

B.  Importance  of  the  task--Remind  your  participants  that: 
--the  army  is  sincerely  interested  in  their  responses 

— every  group  member  is  encouraged  to  make  as  large  a  contribution 

to  the  group  as  possible 
—No  individual  will  be  identified  in  any  way.    They  are  not  to 

sign  any  form  or  in  any  way  identify  themselves  with  this  group. 

Because  they  are  anonymous  participants,  they  are  encouraged  to 

be  as  candid  and  honest  as  possible. 
— When  presenting  possible  solutions,  each  member  is  asked  to  be 

creative  but  also  practical  considering  the  monetary  and 

personnel  constraints  facing  AROTC.    The  ideas  generated  will 

be  used  solely  by  the  local  AROTC  cadre. 

C.  Describe  the  Process--Briefly  (one  minute  maximum)  tell  the 
participants  there  will  be  six  steps  in  the  Nominal  Group  Process: 

1.  The  Silent  Writing  of  ideas  by  each  individual 

2.  The  Listing  of  these  ideas--one  at  a  time 

3.  The  Brief  Discussion  of  each  item 

4.  The  Voti ng--secret  ballot  to  select  the  Top  "7"  most 
important  items 

5.  Discussion  of  the  Vote  (optional  with  each  group) 

6.  Re-Vote  (optional  with  each  group) 

Do  not  discuss  the  process  in  detail--explain  that  each  step  will 
be  described  as  it  occurs. 

Try  to  avoid  answering  questions  about  the  process  at  this  time. 

D.  Other  Items 

--urge  the  group  to  avoid  looking  or  listening  to  other  groups 
near  by 

--ask  them  to  remain  quiet  when  they  are  not  presenting  an  idea 
or  making  a  point 
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NGT  PROCESS 

Step  I     Silent  Generation  of  Ideas  in  Writing 

--note  the  starting  time  somewhere  on  your  summary  voting  sheet 
--start  the  tape  recorder  (if  assigned) 
--verbally  read  the  Nominal  Group  question  to  the  group 
--remind  them  you  want  creative  but  practical  and  useful  solutions 
--ask  them  to  put  their  ideas  in  brief  phrases--not  long  sentences 
--whereever  possible,  ideas  should  be  worded  so  that  an  AROTC 
officer  could  understand  their  meaning 

Silent  Writing  (10  minutes) 

--ask  them  to  reread  the  nominal  question  and  silently  and 
independently  write  each  idea  on  a  separate  3x5  card 
(additional  3x5  cards  are  available  from  the  assistants) 

--give  them  ten  minutes  to  write  their  ideas 

--if  anyone  talks,  sanction  the  activity  by  addressing  comments 
to  the  group  as  a  whole,  not  to  any  individual 

--at  the  conclusion  of  the  ten  minute  period,  announce  the  time 
is  up--but  they  can  (and  should)  continue  to  add  any  new  ideas 
at  any  time. 

Step  II    Round  Robin  Recording  of  Ideas  on  a  Flip  Pad 

--as  a  group  leader  you  must  be  very  careful  to  avoid  any 

discussion  or  clarification  of  items  during  this  phase 
--explain  to  the  group  that  the  purpose  of  this  phase  is  to  map 

the  group's  thinking, 
--explain  that  the  ideas  will  be  presented  (one  at  a  time)  by 

each  group  member.    Each  member  will  place  a  single  3x5  card 

with  an  idea  on  it  in  the  "hat"  each  round, 
--as  the  group  leader  you  will  shuffle  the  cards  and  pick  one 

to  record.    Read  it  aloud  and  write  it  on  the  flip  chart.  Then 

read  the  next  card,  etc.,  until  all  of  the  six  ideas  in  the  hat 

have  been  read. ■ 
--explain  that  anyone  can  "pass"  (decline  to  present  an  idea) 

by  merely  placing  a  blank  3x5  card  in  the  "hat"  for  that  round. 

Also  explain  that  they  can  re-enter  at  any  time. 
— explain  that  it  is  highly  desirable  to  "hitchhike"  on  other 

ideas  and  present  them  during  the  next  pass  of  the  hat. 
--as  a  group  leader  if  you  receive  a  card  with  an  idea  that  is 

unclear  or  too  long  you  must  read  the  idea  aloud  and  suggest 

that  someone  in  the  group  should  resubmit  it  in  a  clearer  form 

during  another  round, 
--explain  that  each  individual  must  decide  if  one  of  their  ideas 

is  a  duplicate  of  one  already  presented.    However,  if  there  is 

some  doubt,  it  is  usually  best  to  present  it. 
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--remember  as  group  leader 

a.  you  must  number  each  item 

b.  do  not  reword  any  item 

c.  do  not  allow  any  discussion  or  clarification  of  items  in 
this  phase 

d.  sanction  any  unnecessary  talking 

e.  write  fast  and  legibly  on  the  flip  chart  pages 

f.  ask  an  assistant  if  you  run  out  of  flip  chart  sheets 
--remind  the  participants  to  use  brief  phrases,  also  note  that 

although  writing  ideas  on  the  flip  charts  will  take  some  time, 
they  should  remember  that  this  extra  time  will  help  to  "gel" 
ideas  they  might  have  in  their  minds. 

Begin  the  Round  Robin  Recording 

--continue  to  pass  the  "hat"  until  at  least  two  rounds  in  which 
all  the  3x5  cards  are  blank. 

Step  III  Serial  Discussion  for  Clarification 

(You  might  check  on  the  tape  recorder  at  this  time) 

— explain  to  the  group  that  the  purpose  of  this  phase  is  to: 

a.  get  any  unclear  items  clarified 

b.  define  the  meaning  of  items 

c.  present  positive  or  negative  evaluative  comments 
--explain  that  final  judgments  will  be  made  by  voting,  so 

arguments  between  members  is  unnecessary 
--dupl icates--note  that  it  is  not  the  goal  of  the  group  to 
combine  or  eliminate  items.    If  however  the  group  feels  that 
items  should  be  eliminated  or  combined,  an  individual  must 
make  a  proposal  outlining  the  change.    If  any  one  member 
objects,  then  the  proposal  fails.    Objections  do  not  have  to 
be  explained. 
--Remember 

a.  pace  the  group--don't  spend  too  much  time  (more  than  2-3 
minutes  as  a  general  rule)  on  any  one  item. 

b.  don't  allow  arguments. 

c.  feel  free  to  skip  over  an  item  if  no  one  volunteers  any 
comments  on  it. 

d.  be  careful  not  to  force  any  member  to  defend  or  clarify 
any  idea  he  or  she  may  have  presented  in  Phase  II. 

--tell  the  participants  they  will  be  asked  to  vote  on  the  top 
"7"  items  in  the    next  phase  so  they  may  want  to  note  any  items 
they  consider  valuable  during  the  discussion  phase. 

--remind  participants  new  items  can  still  be  added  at  any  time 
to  the  list. 

--begin  the  discussion  starting  with  item  #1. 
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Step  IV  Preliminary  Vote  on  Item  Importance 

Purpose--Explain  that  the  group  will  now  secretly  vote  on  the  top 
seven  items  from  the  entire  list. 

--ask  the  participants  to  use  one  of  the  voting  sheets  they 
were  given 

--remind  them  to  assign  7  points  to  the  most  important  item, 

6  to  the  next  most  important,  etc. 
--ask  them  to  reread  the  question  and  then  record  their  top  "7" 

items  using  the  number  assigned  to  it  on  the  flip  chart, 
--voting  is  confidential.    After  a  member  has  voted  he  should 

fold  his  sheet  in  half  and  remain  quiet, 
--collect  the  folded  voting  sheets,  shuffle  them  and  tabulate 

the  votes  on  the  group  leaders  summary  voting  sheet.    Do  this 

on  your  own,  but  as  quickly  as  possible, 
--announce  the  Top  "7"  items,  circle  them  on  the  flip  chart  pages 

and  write  the  number  of  votes  they  received  next  to  them  (on 

the  flip  chart  pages) 

Step  V    Discussion  of  the  Vote  (Optional) 

If  there  are  any  "ties"  on  the  vote,  there  must  be  a  revote  on 
the  tied  items  only.    However,  the  group  can  (at  its  option)  decide 
more  discussion  is  needed  on  any  one  or  several  of  the  items.  If 
this  option  is  chosen,  the  discussion  is  for  clarification  and  evaluati 
comments  only. 

Step  VI    Final  Vote  (Optional) 

--if  a  final  vote  is  desired,  the  second  individual  voting  sheet 
is  used.    The  procedures  outlined  in  Step  IV  are  then  repeated. 

--if  a  final  vote  is  necessary  only  to  break  a  tie  then  each 
individual  assigns  one  point  to  the  preferred  item  and  0  points 
to  the  least  preferred  item  (a  2-1-0  system  is  used  for  3  way 
ties).    The  back  of  a  voting  sheet  can  be  used  to  vote  on  "ties". 

--notify  the  group  of  the  results  of  the  final  vote. 


END  OF  REGULAR  SESSION 

—  Note  the  time,  and  mark  it  and  the  total  elapsed  time  on  the 

final  voting  sheet. 
--Turn  off  the  tape  recorder. 

Additional  Information 

--remind  the  participants  that  the  top  "7"  items  will  be 
presented  to  the  AROTC  Cadre  whom  might  not  understand  your  exact 
meaning.    If  the  group  chooses,  it  can  add  a  short  explanation 
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of  any  item  (not  to  exceed  a  short  paragraph).    If  the 
group  so  chooses,  add  that  explanation  (citing  the  item 
number)  to  your  group  leaders  summary  voting  sheet  (page 
2). 

Satisfaction  Questionnaire 

— give  each  participant  one  questionnaire.    Ask  him  to  read  it 

carefully  and  answer  each  question, 
--when  he  has  completed  his  questionnaire,  ask  him  to  turn  it 

over  and  wait  for  the  others  to  finish.    (Then  collect  the 

questionnai  res) 

Final  Steps 

--ask  each  participant  to  hand  in  all  of  his  remaining  3x5 
cards  (whether  they  have  ideas  on  them  or  not).  You  should 
shuffle  these,  remove  any  cards  with  any  ideas  on  them  and 
put  them  in  a  stack  with  a  rubber  band  around  them.  Label 
the  stack  "unpresented  ideas". 

--collect  the  pencils  and  any  remaining  items  from  the 
participants. 

--thank  them  for  their  assistance  and  ask  them  to  get  up  quietly 
and  leave  the  area,  so  that  the  other  groups  may  finish. 

FINAL  TASKS  FOR  THE  GROUP  LEADER 

--be  sure  to  note  the  elapsed  time  on  the  second  page  of  the 

summary  voting  sheet, 
--write  the  top  seven  items  on  the  summary  voting  sheet  (page  2) 

and  any  explanatory  comments  the  group  chose  to  add.  Remember 

to  write  the  items  out  completely,  not  just  the  item  number, 
--make  a  note  on  the  summary  voting  sheet  of  the  total  number  of 

items  generated  by  the  group  (total  number--less  any  eliminated 

i  tems ) , 

--if  you  have  any  final  comments  or  anything  went  wrong  during 
the  session  explain  that  on  the  summary  voting  sheet  also. 

--take  the  cassette  (tape)  out  of  the  recorder  and  put  it  in  the 
group  leaders  envelope. 

--put  all  pencils,  markers,  voting  sheets,  etc.  in  the  group  leaders 
envelope. 

--unplug  the  tape  recorder  and  turn  it  and  the  group  leaders 

envelope  into  one  of  the  assistants. 
— pat  yourself  on  the  back  for  a  job  well  done  and  add  a  thank  you 

from  AROTC  and  the  Management  Department 


Group  Number 


APPENDIX  C 


DELBECQ  NGT--INSTRUCTIONS  TO  GROUP  LEADERS 


Please  read  these  instructions  carefully.  If  you  have  any  questions, 
ask  one  of  the  assistants  to  help  you. 

Your  Function 

Your  job  as  group  leader  is  to  ensure  that  the  procedures 

(as  outlined  below)  for  the  Nominal  Group  Process  are  strictly 

adhered  to. 


Some  Do's  and  Don'ts 


Don't 

— contribute  your  own  ideas  or  solutions 
--criticize  ideas 
--al low  arguments 

--ask  the  initiator  of  an  idea  to  explain  it 
--suggest  compromises 
--reword  ideas  for  members 


Do 

--try  to  keep  the  process  from  bogging  down 
--write  swiftly  but  legibly  on  the  flip  chart  pages 
--tally  the  votes  swiftly 

--avoid  long  explanations  of  the  Nominal  Group  Process 
--try  to  stop  any  side  conversations  between  members 
--ask  for  assistance  if  you  have  any  questions 


Procedures  for  a  Group  Leader 
Set  Up 

A.  You  will  be  randomly  assigned  6  participants.    If  you  do  not  have 
six,  ask  for  assistance. 

B.  You  will  be  assigned  a  table  either  in  the  mess  hall  or  in  the  rec. 
hall  (to  the  east  of  the  mess  hall  towards  the  river).  Participants 
should  sit  in  a  "U"  shape  (three  to  a  side)  with  you  at  the  head  of 
the  table  with  the  large  recording  sheets. 

C.  Plug  in  the  tape  recorder  (if  assigned)  make  sure  it  works  and 
that  it  is  as  inconspicuous  as  possible. 
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D.    Hand  out  from  your  packet  to  each  participant 
1  pencil 

1  sheet  with  the  topic  question  typed  on  it 

2  individual  voting  sheets 


INTRODUCTION  AND  WELCOME 

A.  Purpose--In  friendly  terms  explain  that:    "You  are  participating 
in  a  dual  purpose  exercise.    The  first  is  to  provide  the  AROTC 
with  honest  feedback  from  its  cadets  on  the  topic  area  listed  on 
the  sheet  you  were  just  handed.    The  army  is  looking  for  creative 
but  practical  solutions  to  this  problem.    The  second  purpose  of 
this  exercise  is  to  acquaint  you  with  a  useful  leadership  tool 

for  group  decision  making--the  Nominal  Group  Technique.    As  a  future 
leader  you  may  find  that  this  technique  might  have  applications 
in  many  facets  of  army  life." 

B.  Importance  of  the  task--Remind  your  participants  that: 
— the  army  is  sincerely  interested  in  their  responses 

— every  member  is  encouraged  to  make  as  large  a  contribution  to 

the  group  as  possible. 
--No  individual  will  be  identified  in  any  way.    They  are  not  to 
.  sign  any  form  or  in  any  way  identify  themselves  with  this  group. 

Because  they  are  anonymous  participants,  they  are  encouraged  to 

be  as  candid  and  honest  as  possible. 
--When  presenting  possible  solutions,  each  member  is  asked  to  be 

creative  but  also  practical  considering  the  monetary  and 

personnel  constraints  facing  AROTC.    The  ideas  generated  will 

be  used  solely  by  the  local  AROTC  cadre. 

C.  Describe  the  Process--Briefly  (one  minute  maximum)  tell  the 
participants  there  will  be  six  steps  in  the  Nominal  Group  Process: 

1.  The  Silent  Writing  of  ideas  by  each  individual. 

2.  The  Listing  of  these  ideas--one  at  a  time. 

3.  The  Brief  Discussion  of  each  item. 

4.  The  Voting--secret  ballot  to  select  the  Top  "7"  most  important 
i  terns . 

5.  Discussion  of  the  Vote  (optional  with  each  group). 

6.  Re-Vote  (optional  with  each  group). 

D.  Other  Items 

--urge  the  group  to  avoid  looking  or  listening  to  other  groups 
near  by. 

--ask  them  to  remain  quiet  when  they  are  not  presenting  an  idea 
or  making  a  point. 
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NGT  PROCESS 

Step  I    Silent  Generation  of  Ideas  in  Writing 

--note  the  starting  time  somewhere  on  your  summary  voting  sheet 
--start  the  tape  recorder  (if  assigned) 
--verbally  read  the  Nominal  Group  question  to  the  group 
--remind  them  you  want  creative  but  practical  and  useful  solutions 
--ask  them  to  put  their  ideas  in  brief  phrases--not  long  sentences 
--whereever  possible,  ideas  should  be  worded  so  that  an  AROTC 
officer  could  understand  their  meaning 

Silent  Writing  (10  minutes) 

--ask  them  to  reread  the  nominal  question  and  silently  and 

independently  write  their  ideas  down  in  any  order  on  the  nominal 
question  sheet 

— give  them  ten  minutes  to  write  their  ideas 

--if  anyone  talks,  sanction  the  activity  by  addressing  comments  to 

the  group  as  a  whole,  not  to  any  individual 
--at  the  conclusion  of  the  ten  minute  period,  announce  the  time  is 

up--but  they  can  (and  should)  continue  to  add  any  new  ideas  at 

any  time. 

Step  II    Round  Robin  Recording  of  Ideas  on  a  Flip  Pad 

--as  a  group  leader  you  must  be  very  careful  to  avoid  any  discussion 

or  clarification  of  items  during  this  phase, 
--explain  to  the  group  that  the  purpose  of  this  phase  is  to  map 

the  group's  thinking, 
--explain  that  ideas  will  be  presented  (one  at  a  time)  by  each 

group  member  in  a  round  robin  fashion.    After  each  member  has 

presented  an  idea,  the  round  robin  starts  again, 
--explain  that  any  one  can  "pass"  (decline  to  present  an  idea) 

by  merely  raising  his/her  hand  if  he/she  has  no  further  items 

to  present  at  that  time.    Also  explain  that  they  can  re-enter 

at  any  time. 

--explain  that  it  is  highly  desirable  to  "hitchhike"  on  other 

ideas  and  present  them  during  your  turn, 
--explain  that  each  individual  must  decide  if  one  of  their  ideas 

is  a  duplicate  of  one  already  presented.    However,  if  there  is 

some  doubt  it  is  usually  best  to  present  it. 
--remember  as  group  leader 

a.  you  must  number  each  item 

b.  do  not  reword  any  item 

c.  do  not  al low  any  discussion  or  clarification  of  items  in 
this  phase 

d.  sanction  any  unnecessary  talking 

e.  write  fast  and  legibly  on  the  flip  chart  pages 

f.  ask  an  assistant  if  you  run  out  of  flip  chart  sheets 
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--remind  the  participants  to  use  brief  phrases,  also  note 

that  although  writing  ideas  on  the  flip  charts  will  take  some 
time,  they  should  remember  that  this  extra  time  will  help  to 
"gel"  ideas  they  might  have  in  their  minds. 

Begin  the  Round  Robin  Recording 

--continue  recording  until  everyone  "passes"  on  the  same  round 

Step  III    Serial  Discussion  for  Clarification 

(You  might  check  on  the  tape  recorder  at  this  time) 

--explain  to  the  group  that  the  purpose  of  this  phase  is  to: 

a.  get  any  unclear  items  clarified 

b.  define  the  meaning  of  items 

c.  present  positive  or  negative  evaluative  comments 
--explain  that  final  judgments  will  be  made  by  voting,  so 

arguments  between  menbers  is  unnecessary 
--duplicates--note  that  it  is  not  the  goal  of  the  group  to 

combine  or  eliminate  items.    If,  however,  the  group  feels  that 
items  should  be  eliminated  or  combined,  an  individual  must  make 
a  proposal  outlining  the  change.    If  any  one  member  objects, 
then  the  proposal  fails.    Objections  do  not^  have  to  be  explained. 
-^-Remember 

a.  pace  the  group--don't  spend  too  much  time  (more  than  2-3 
minutes  as  a  general  rule)  on  any  one  item. 

b.  don't  allow  arguments. 

c.  feel  free  to  skip  over  an  item  if  no  one  volunteers  any 
comments  on  it. 

d.  be  careful  not  to  force  any  member  to  defend  or  clarify 
any  idea  he  or  she  may  have  presented  in  Phase  II. 

--tell  the  participants  they  will  be  asked  to  vote  on  the  top  "7" 
items  in  the  next  phase  so  they  may  want  to  note  any  items  they 
consider  valuable  during  the  discussion  phase. 

--remind  the  participants  new  items  can  still  be  added  at  any  time 
to  the  list. 

--begin  the  discussion  starting  with  item  #1. 

Step  IV    Preliminary  Vote  on  Item  Importance 

Purpose--Explain  that  the  group  will  now  secretly  vote  on  the  top 
seven  items  from  the  entire  list. 

--ask  the  participants  to  use  one  of  the  voting  sheets  they  were 
gi  ven 

--remind  them  to  assign  7  points  to  the  most  important  item,  6 

to  the  next  most  important,  etc. 
--ask  them  to  reread  the  question  and  then  record  their  top  "7" 

items  using  the  number  assigned  to  it  on  the  flip  chart, 
--voting  is  confidential.    After  a  member  has  voted  he  should  fold 

his  sheet  in  half  and  remain  quiet. 
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--collect  the  folded  voting  sheets,  shuffle  them  and  tabulate 

the  votes  on  the  group  leaders  summary  voting  sheet.    Do  this 

on  your  own,  but  as  quickly  as  possible, 
--announce  the  Top  "7"  items,  circle  them  on  the  flip  chart 

pages  and  write  the  number  of  votes  they  received  next  to  them 

(on  the  flip  chart  pages) 

Step  V    Discussion  of  the  Vote  (Optional) 

If  there  are  any  "ties"  on  the  vote,  there  must  be  a  revote  on 
the  tied  items  only.    However,  the  group  can  (at  its  option)  decide 
more  discussion  is  needed  on  any  one  or  several  of  the  items.    If  this 
option  is  chosen,  the  discussion  is  for  clarification  and  evaluative 
comments  only. 

Step  VI    Final  Vote  (Optional) 

--if  a  final  vote  is  desired,  the  second  individual  voting  sheet 
is  used.    The  procedures  outlined  in  Step  IV  are  then  repeated. 

--if  a  final  vote  is  necessary  only  to  break  a  tie  then  each 
individual  assigns  one  point  to  the  preferred  item  and  0  points 
to  the  least  preferred  item  (a  2-1-0  system  is  used  for  3  way 
ties).    The  back  of  a  voting  sheet  can  be  used  to  vote  on 
"ties". 

—  notify  the  group  of  the  results  of  the  final  vote. 

END  OF  REGULAR  SESSION 

--Note  the  time,  and  mark  it  and  the  total  elapsed  time  on  the 

final  voting  sheet. 
--Turn  off  the  tape  recorder. 

Additional  Information 

--remind  the  participants  that  the  top  "7"  items  will  be  presented 
to  the  AROTC  Cadre  whom  might  not  understand  your  exact  meaning. 
If  the  group  chooses,  it  can  add  a  short  explanation  of  any  item 
(not  to  exceed  a  short  paragraph).  If  the  group  so  chooses,  add 
that  explanation  (citing  the  item  number)  to  your  group  leaders 
summary  voting  sheet  (page  2). 

Satisfaction  Questionnaire 

--give  each  participant  one  questionnaire.    Ask  him  to  read  it 

carefully  and  answer  each  question, 
--when  he  has  completed  his  questionnaire,  ask  him  to  turn  it  over 

and  wait  for  the  others  to  finish.    (Then  collect  the  questionnai 
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Final  Steps 

--ask  each  participant  to  hand  in  his/her  sheet  with  the  Nominal 
Question  on  it  as  well  as  the  items  he/she  presented  to  the  group 
Shuffle  the  sheets.    These  sheets  will  remain  with  the  project 
leader  and  they  will  never  be  seen  by  the  AROTC 
--collect  the  pencils  and  any  remaining  items  from  the  participants 
--thank  them  for  their  assistance  and  ask  them  to  get  up  quietly 
and  leave  the  area,  so  that  the  other  groups  may  finish. 

FINAL  TASKS  FOR  THE  GROUP  LEADER 

--be  sure  to  note  the  elapsed  time  on  the  second  page  of  the 

summary  voting  sheet, 
--write  the  top  seven  items  on  the  summary  voting  sheet  (page  2) 

and  any  explanatory  comments  the  group  chose  to  add.  Remember 

to  write  the  items  out  completely,  not  just  the  item  number, 
--make  a  note  on  the  summary  voting  sheet  of  the  total  number  of 

items  generated  by  the  group  (total  number--less  any  eliminated 

i  tems ) . 

--if  you  have  any  final  comments  or  anything  went  wrong  during 

the  session  explain  that  on  the  summary  voting  sheet  also. 
—  take  the  cassette  (tape)  out  of  the  recorder  and  put  it  in  the 

group  leaders  envelope. 
— put  all  pencils,  markers,  voting  sheets,  etc.  in  the  group 

leaders  envelope, 
--unplug  the  tape  recorder  and  turn  it  and  the  group  leaders 

envelope  into  one  of  the  assistants, 
--pat  yourself  on  the  back  for  a  job  well  done  and  add  a  thank  you 

from  AROTC  and  the  Management  Department. 


Group  Number 


APPENDIX  D 
PARTICIPANT  SATISFACTION  QUESTIONNAIRE 

The  following  questionnaire  is  designed  to  assist  us  in  under- 
standing how  satisfied  you  are  with  the  Nominal  Group  meeting  you  just 
completed. 

—  Please  take  your  time  and  answer  as  accurately  as  possible. 
--Please  answer  every  question. 

--Please  circle  the  appropriate  number  for  each  question 

(Example:    Totally  Totally 

Unsatisfied  /0/l/2/3/4/5/6/lg)/8/9/10/Satisfied) 

Indif. 

Part  I.    GROUP  PROCESS  -  Satisfaction  with  different  aspects  of  the 

Nominal  Group  Process 

1.  How  satisfied  are  you  that  the  ideas  presented  to  the  group  could 
"ot  be  "traced"  back  to  the  individuals  who  presented  them? 

Totally  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indifferent 

2.  How  satisfied  are  you  that  you  were  free  to  refuse  to  defend  or 
clarify  ideas  you  submitted  to  the  group? 

Totally  _  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indifferent 

3.  How  satisfied  are  you  that  you  were  free  to  "pass"  without 
embarassment  during  the  Round  Robin  presentation  of  ideas? 

Totally  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indi  fferent 

4.  How  satisfied  are  you  that  you  were  free  to  challenge  or 
disagree  with  an  item  during  the  discussion  phase? 

Totally  Totally 
Unsatsified  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indifferent 
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5.  How  satisfied  are  you  that  you  were  free  to  ask  questions  and 
seek  clarification  of  items  during  the  discussion  phase? 

Totally  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indifferent 

6.  How  satisfied  are  you  that  the  ideas  you  presented  to  the  group 
were  given  fair  consideration  by  the  group? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indi  fferent 

7.  How  satisfied  are  you  with  the  opportunities  you  had  to  present 
your  ideas  to  the  group? 

Totally  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisifed 

Indi  fferent 

8.  How  satisfied  are  you  that  the  group  process  was  time  efficient? 

Totally  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indi  fferent 

9.  How  satisfied  are  you  that  the  group  process  was  not  dominated 
by  any  one  individual  or  small  group? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indifferent 

10.  How  satisfied  are  you  with  the  performance  of  the  group  leader? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indi  fferent 

11.  Overall,  how  satisfied  are  you  with  all  of  the  procedural  aspects 
of  the  nominal  group  process? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indi  fferent 
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Part  II    GROUP  OUTPUT  -  Satisfaction  with  the  final  output  of  the  group 

12.  How  satisfied  are  you  with  the  quantity  (number  of  items)  generated 
by  the  group? 

Totally  Totally 
Unsatisfied  /  0  /  1  /  2  /  3  /  4  /  5  /  6  /  7  /  8  /  9  /  10  /  Satisfied 

Indifferent 

13.  How  satisfied  are  you  with  the  quality  of  the  items  generated 
by  the  group? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indifferent 

14.  How  satisfied  are  you  that  the  items  generated  by  your  group 
are  a  fair  representation  of  the  group's  feeling? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indifferent 

15.  How  satisfied  are  you  with  the  prospect  of  presenting  the 
group's  output  as  your  position? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indifferent 

16.  How  satisfied  are  you  that  the  group  attained  its  over-all  goals? 

Totally  Totally 
Unsatisfied  /0/1/2/3/4/5/6/7/8/9/10/  Satisfied 

Indi  fferent 

Part  III  SHORT  ANSWER  QUESTIONS 

17.  What  aspects  of  the  process  did  you  like  the  most? 


18.    What  aspects  of  the  process  did  you  like  the  least? 


19.    Do  you  have  any  suggestions  for  the  improvement  of  the  process? 
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20.  Do  you  have  any  general  comments  about  the  process,  the 
recorder  or  the  experiment? 

Part  IV 

Have  you  ever  participated  in  a  decision  meeting  or  conference 
that  used  a  process  other  than  the  Nominal  Group  Technique  you  just 
completed?    If  so,  please  compare  this  "other"  process  with  the  Nominal 
Group  Technique.    (Circle  one  for  each  question) 

21.  Comparing  Group  Leaders,  in  the  "other"  process  the  leader  was 

1  Much  More     2  More  3  About  the     4  More  5  Much  More 

Dominant         Dominant        Same  Submissive  Submissive 

22.  Comparing  the  amount  of  discussion,  in  the  "other"  process  there  was 

1  Much  More     2  More  3  About  the     4  Less  5  Much  Less 

Discussion       Discussion     Same  Discussion  Discussion 

23.  Comparing  the  amount  of  arguing,  in  the  "other"  process  there  was 

1  Much  More     2  More  3  About  the     4  Less  5  Much  Less 

Same 

24.  Comparing  how  a  solution  was  chosen,  in  the  "other"  process  the 
solution  was  generally  selected  by 

1  Unanimous     2  Majority     3  By  the  4  Generally  no  agreement 

Vote  Vote  Group  Leader      was  reached 

25.  Comparing  how  free  you  felt  to  express  ideas,  in  the  "other" 
process  you  felt 

1  Much  More     2  More  '  3  About  the     4  Less        5  Much  Less 

Free               Free  Same               Free  Free 

26.  In  general,  did  you  find  the  Nominal  Group  Process  more  effective 
or  satisfying  than  these  "other  processes  you  participated  in? 

1  Yes        2  No 


WHEN  YOU  COMPLETE  THIS  FORM,  PLEASE  REMAIN  QUIET  UNTIL  THE  GROUP  LEADER 
COLLECTS  THE  FORMS. 


10/24/78 
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APPENDIX  E 

INSTRUCTIONS  FOR  CONSENSUAL  JUDGEMENT  RATERS 


Please  read  Question  "A"  carefully— (you  may  reread  it  as  many  times 
as  necessary  during  this  session). 

Your  job  is  to  evaluate  the  relative  quality  of  the  responses  made  by 
the  AROTC  cadets  to  the  question  you  have  just  read. 

You  should  first  find  a  quiet  place  with  a  large  table  so  that  you 
can  spread  the  cards  out  easily. 

Step  1        Scan  through  all  the  "A"  cards  quickly  to  get  an  idea 
of  what  ideas  have  been  generated. 

Step  2        From  the  cards  pick  the  ones  you  consider  to  be  obvious 
duplicates  and  paper  clip  them  together. 

CRITERIA  FOR  SCORING 

You  are  now  to  begin  the  actual  "quality"  or  subjective  rating  of  items. 

You  are  to  rank  items  which  (in  your  opinion)  are  the  highest  and 
lowest  quality  solutions  or  responses  to  Question  "A".  Your  score  for 
each  item  should  reflect  the  quality  of  that  item  based  on  its 
feasibility,  importance  and/or  merit. 

Very  Highest  -  Step  3    Select  the  items  which  (in  your  opinion)  are  the 

very  highest  quality  responses.    Group  them  in  a 
stack.    This  stack  may  contain  only  one  or  several 
items.    Your  only  criteria  for  inclusion  is  that 
they  are  generally  all  of  the  same  average  qual ity 
and  thus  belong  together  (even  if  they  cover 
different  topics).    You  do  not  have  to  rate  the 
items  wi thin  this  stack  of  cards. 

Very  Lowest  -  Step  4     Next,  select  the  items  which  (in  your  opinion) 

are  the  very  lowest  quality  responses  to  this 
question.    Group  them  together  in  a  stack  using 
the  same  criteria  of  similar  average  quality. 
This  stack  may  contain  a  larger  or  smaller  number 
of  cards  than  the  previous  stack. 

2nd  Highest  -  Step  4     From  the  ramining  cards  select  the  items  which 

you  consider  the  next  highest  in  over  all  quality 
as  you  did  in  Step  4. 
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2nd  Lowest  -  Step  6       From  the  remaining  cards  select  the  items  (card) 

which  you  consider  the  next  lowest  in  over-all 
quality. 

Step  7  Continue  selecting  the  next  highest  quality 

items/ then  the  next  lowest  quality  until  you 
feel  all  the  remaining  cards  are  of  the  same 
qual ity . 

Step  8  You  may  (if  you  wish)  reconsider  moving  any  item 

to  another  stack  of  higher  or  lower  quality. 

Step  9  Now  on  the  "A"  Rating  Sheet  (provided)  note  the 

number  of  stacks  you  ended  up  with. 

Step  10  Write  the  number  of  each  of  the  items  in  the 

Very  Highest  Quality  group  (stack)  on  the  1st 
line  of  the  "A"  Rating  Sheet  (the  number  of  the 
item  is  in  red;  also,  the  order  in  which  you  list 
the  items  within  the  group  is  not  important). 

Step  11  Write  the  number  of  each  of  the  items  in  the  Very 

Lowest  Quality  stack  on  the  bottom  line  on  the 
Rating  Sheet. 

Step  12  Continue  listing  the  items  by  group  (Highest/ 

Lowest)  until  you  only  have  one  stack  left. 
Place  the  number  of  each  item  in  that  last 
stack  on  the  middle  line  of  the  "A"  Rating  Sheet. 


You  are  now  finished  with  Question  A  -  remove  all  clips,  shuffle  the 
cards  briefly  and  put  a  rubber  band  around  all  of  the  "A"  cards  and 
the  Question  "A"  card. 

Step  13  Read  Question  "B".    Take  the  "B"  cards  and 

briefly  scan  through  them. 

Step  14  Now  repeat  Steps  2  through  12  using  the  "B"  cards. 

Question  "B"  and  the  "B"  Rating  Sheet. 

When  you  finish  completing  the  "B"  Rating  Sheet,  reshuffle  and  rubber 
band  the  "B"  cards  and  sign  both  rating  sheets. 

Return  all  the  cards  and  the  two  sheets  ("A"  and  "B")  to  Captain  Knowlton. 


Thank  you  very  much  for  your  assistance. 


John  Sullivan 
Department  of  Management 
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